Stata to Python Equivalents

1   Intro/Note on Notation

Coding in Python is a little different than coding in Stata.

In Stata, you have one dataset in memory. The dataset is a matrix where each column is a "variable" with a unique name and each row has a number (the special variable _n). Everything in Stata is built around this paradigm.

Python is a general purpose programming language where a "variable" is not a column of data. Variables can be anything, a single number, a matrix, a list, a string, etc. The Pandas package implements a kind of variable called a DataFrame that acts a lot like the single dataset in Stata. It is a matrix where each column and each row has a name. The key distinction in Python is that a DataFrame is itself a variable and you can work with any number of DataFrames at one time. You can think of each column in a DataFrame as a variable just like in Stata, except that when you reference a column, you also have to specify the DataFrame.

The Stata-to-Python translations below are written assuming that you have a single DataFrame called df. Placeholders like <varname> and <dtafile> show where user-specified values go in each language. Note that in many cases, <varname> will be simple text in Stata (e.g., avg_income) while in Python it will be a string ('avg_income'). If you were to write df[avg_income] without quotes, Python would go looking for a variable--a list, a number, a string--that's been defined somewhere else. Because of this, <varlist> in Python represents a list of variable names: ['educ', 'income', 2017].

2   Note on composability

One concept that will push your Python skills forward quickly is "composability." That is, you can combine base-level commands to "create" whole new commands. This is a little more difficult in Stata where each line of code acts on an entire dataset.

For example, let's say you know two commands in Python/pandas. First, df[<condition>], which returns the rows of DataFrame df for which the Boolean <condition> is True. This is like keep if <condition> in Stata, except df[<condition>] is itself a DataFrame that can be acted upon independently of df itself. This means that you now know the general if <condition> syntax for every other pandas command.

So let's say the second command you know is df.describe() which is the equivalent of Stata's summary. This command doesn't have it's own dedicated if command. It doesn't need one. All you need is df[<condition>].describe().

In this way, you can create (i.e., "compose") new commands. You may notice that sometimes Python/pandas will require composed commands where Stata uses a one-liner (for example, see Stata's drop varstem* below). While it may may annoying to type a few more characters, this is actually a good thing! It means that there are fewer base commands to learn in pandas. It's like learning an alphabet with 26 letters and composing millions of words instead of learning millions of individual hieroglyphs.

This also means that sometimes there is more than one way to do things. Sometimes, it doesn't matter which way you do it. Other times, one way of doing things will execute more quickly. You can use the %timeit magic command in IPython to easily test which one is faster. But this will usually only an issue with with larger data sets (usually several gigabytes). For example, in large Groupby operations .shift(fillna=0) is much, much slower than shift().fillna(0) even though the end result is the same.

3   Input/Output

4   Sample Selection

5   Data Info and Summary Statistics

df.groupby(['var1', 'var2'])
  .agg(<func>)
  .unstack('var2')

6   Variable Manipulation

Python doesn't have "labels" built into DataFrames like Stata does. However, you can use a dictionary to map data values to labels when necessary.

variable_labels = {
    1: "First Category",
    2: "Second Category",
    3: "Last Category",
}

7   Bysort

All bysort and egen commands will be compositions using groupby.agg or groupby.transform

df['group_sum'] = (
    df
    .sort_values(['group_var1', 'group_var2', 'sort_var1'])
    .groupby(['var1', 'var2'])
    .apply(lambda x: x.iloc[j, :])
)

df['jth_val'] = (
    df
    .sort_values(['group_var1', 'group_var2', 'sort_var1'])
    .groupby(['var1', 'var2'])['var']
    .transform(lambda x: x.iloc[j])
)

Note that groupby was also used above to accomplish a collapse. Like, a Stata Collapse, a pandas Groupby reduces the size of the data set. If you want to instead execute what's called a window function in SQL, which does not reduce the shape of the original data set, you can call df.groupby.transform(<function>) instead of df.groupby.agg(<function>).

If for some reason transform doesn't work in your use case, you can also re-join the collapsed data back onto your original data set:

df = pd.read_csv('my_data.csv')
state_pop_sum = df.groupby('state')['population'].sum()
df = df.join(state_pop_sum.to_frame('state_pop_sum'),     # Give the new column a name besides 'population'
             on='state')

8   Panel Data

There is no general equivalent to tsset in Python. However, you can accomplish most if not all of the same tasks using a DataFrame's index (the row's equivalent of columns.) In Stata, the "DataFrame" in memory always has the observation row number, denoted by the Stata built-in variable _n. In Python and Pandas, a DataFrame index can be anything (though you can also refer to rows by the row number; see .loc vs iloc). It can also be hierarchical with mutiple levels. It is a much more general tool than tsset.

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: df0 = pd.DataFrame({'var1': np.arange(6),
   ...:                     'id': [1, 1, 2, 2, 3, 3],
   ...:                     'period': [0, 1] * 3})

In [4]: print(df0)
   var1  id  period
0     0   1       0
1     1   1       1
2     2   2       0
3     3   2       1
4     4   3       0
5     5   3       1

In [5]: df = df0.set_index(['id', 'period']).sort_index()

In [6]: print(df)
           var1
id period
1  0          0
   1          1
2  0          2
   1          3
3  0          4
   1          5

In [7]: df['var1_lag'] = df.groupby(level='id')['var1'].shift()

In [8]: print(df)
           var1  var1_lag
id period
1  0          0       NaN
   1          1       0.0
2  0          2       NaN
   1          3       2.0
3  0          4       NaN
   1          5       4.0

In [9]: df['var1_for'] = df.groupby(level='id')['var1'].shift(-1)

In [10]: print(df)
           var1  var1_lag  var1_for
id period
1  0          0       NaN       1.0
   1          1       0.0       NaN
2  0          2       NaN       3.0
   1          3       2.0       NaN
3  0          4       NaN       5.0
   1          5       4.0       NaN

9   Merging and Joining

NOTE: Merging in Python is like R, SQL, etc. See additional explanation below.

Merging with Pandas DataFrames does not require you to specify "many-to-one" or "one-to-many". Pandas will figure that out based on whether the variables you're merging on are unique or not. However, you can specify what sub-sample of the merge to keep using the keyword argument how, e.g., df_joint = df1.join(df2, how='left') is the default for join while how='inner' is the default for pd.merge.

10   Reshape

Like with merging, reshaping a DataFrame in Python is a bit different because of the paradigm shift from the "only one data table in memory" model of Stata to "a data table is just another object/variable" of Python. But this difference also makes reshaping a little easier in Python.

The most fundamental reshape commands in Python/pandas are stack and unstack:

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: long = pd.DataFrame(np.arange(8),
   ...:                     columns=['some_variable'],
   ...:                     index=pd.MultiIndex.from_tuples(
   ...:                         [('a', 1), ('a', 2),
   ...:                          ('b', 1), ('b', 2),
   ...:                          ('c', 1), ('c', 2),
   ...:                          ('d', 1), ('d', 2)]))

In [4]: long.index.names=['unit_id', 'time']

In [5]: long.columns.name = 'varname'

In [6]: long
Out[6]:
varname       some_variable
unit_id time
a       1                 0
        2                 1
b       1                 2
        2                 3
c       1                 4
        2                 5
d       1                 6
        2                 7

In [7]: wide = long.unstack('time')

In [8]: wide
Out[8]:
varname some_variable
time                1  2
unit_id
a                   0  1
b                   2  3
c                   4  5
d                   6  7

In [9]: long2 = wide.stack('time')

In [10]: long2
Out[10]:
varname       some_variable
unit_id time
a       1                 0
        2                 1
b       1                 2
        2                 3
c       1                 4
        2                 5
d       1                 6
        2                 7

Here Input 3 creates a DataFrame, Input 4 gives each of the index columns a name, and Input 5 names the columns. Coming from Stata, it's a little weird to think of the column names themselves having a "name", but the columns names are just an index like the row names are. It starts to make more sense when you realize columns don't have to be strings. They can be integers, like years or FIPS codes. In those cases, it makes a lot of sense to give the columns a name so you know what you're dealing with.

Input 6 does the reshaping using unstack('time'), which takes the index 'time' and creates a new column for every unique value it has. Notice that the columns now have multiple levels, just like the index previously did. This is another good reason to label your index and columns. If you want to access either of those columns, you can do so as usual, using a tuple to differentiate between the two levels:

In [11]: wide[('some_variable', 1)]
Out[11]:
unit_id
a    0
b    2
c    4
d    6
Name: (some_variable, 1), dtype: int32

If you want to combine the two levels (like Stata defaults to), you can simply rename the columns:

In [13]: wide_single_level_column = wide.copy()

In [14]: wide_single_level_column.columns = [
    ...:        '{}_{}'.format(var, time)
    ...:        for var, time in wide_single_level_column.columns]

In [15]: wide_single_level_column
Out[15]:
                  some_variable_1  some_variable_2
         unit_id
         a                      0                1
         b                      2                3
         c                      4                5
         d                      6                7

The pivot command can also be useful, but it's a bit more complicated than stack and unstack and is better to revisit pivot after you are comfortable working with DataFrame indexes and columns.

11   Econometrics

12   Plotting

Visualizations are best handled by the packages Matplotlib and Seaborn.

13   Other differences