Enron the smartest guys in the room script

#Enron the smartest guys in the room script code
#Enron the smartest guys in the room script download

The next step is to fit a regression model to see if the word frequency is increasing over time. Word_month_counts % filter(percent % filter( ! str_detect(word, 'font|align|color|image|serif|arial|align|helvetica|padding|link|verdana|fff|space|width|span|spacing|script|servlet|size|email|http')) Word_month_counts % filter(month %within% ( '' %-% '')) %>% filter(word_total >= 500, str_length(word) > 4) %>% count(word, month) %>% complete(word, month, fill = list(n = 0)) %>% inner_join(month_count, by = 'month') %>% mutate(percent = n / month_total) %>% mutate(year = year(month) + yday(month) / 365) Month_count % filter(month %within% ( '' %-% '')) I filtered out words with special characters, and words with 15 or more letters, since those are usually just long strings of nonsense (forwarded emails, etc.). I then did a word count to look for superfluous words that I could exclude to maybe cut down further on the size of the dataset.

#Enron the smartest guys in the room script code

The stop words dataset is built in to the tidytext package, so you donвЂ™t have to do anything special if you have that package loaded, just use it in your code and it will show up. Then unnest to words (one word per row), filter by letters only again, and use anti-join to remove stop words. Using the sentences dataset, I first filtered out all of the вЂњsentencesвЂќ (rows) that had no letters (only numbers or special characters, blank spaces). The message body column is unnested into sentences so that you can perform most tidy text mining operations on it by unnesting the sentences further (into words, n-grams, etc.). To save you the trouble of doing all that, IвЂ™ve uploaded in the final tidy-ish format dataset here.

RDS format using the save_mail function map(files2, save_mail) #get paths for the level one folders (one per person) Tidy_mail % filter(header_flag = 1) %>% group_by(source) %>% summarise(message_body = list(data)) %>% right_join(tidy_mail) Tidy_mail % ungroup() %>% filter( ! is.na(new_col)) %>% mutate(data = str_trim( str_remove(data, new_col), side = 'both')) %>% select( -header_flag) %>% pivot_wider(names_from = new_col, values_from = data) %>% Temp % filter(header_flag = 0) %>% mutate( id just tells me which file the data came from, which is going to be helpful for cleaning.Дf % mutate(header_flag = ifelse( str_detect( lag(data), 'X-FileName:'), 1, 0)) %>% mutate(header_flag = replace_na(header_flag, 0)) %>% group_by(source) %>% mutate(header_flag = cumsum(header_flag)) %>% ungroup() I know that if I read them in as a csv, there will only be one column and all the data will be stored in the rows, so IвЂ™m naming the column вЂњdataвЂќ so that they will all stack on top of each other and I donвЂ™t end up with 557 columns. Otherwise IвЂ™ll be missing some of the data (like the email date). Then I can use map_dfr to read all of the emails into one csv all at one time, using read_delim because I want to use tab instead of comma as the delimiter. The very last person in the list has 557 emails to read in, which is not many compared to some of the other employees. Using recursive = TRUE will give the paths to all the subfolders, and full.names = TRUE will help to read in the data using map_dfr.ДЄfter getting the list, IвЂ™m subsetting it to only include one personвЂ™s mailbox for the purposes of this example.

Getting and Cleaning Messy Text DataДЇirst get a list of all the files in the mail directory folder. If youвЂ™re trying to work with the data and having any issues, just message me and IвЂ™ll try to help.

#Enron the smartest guys in the room script download

You can download the original data at the above link, but in its current format itвЂ™s not super usable, so IвЂ™m going to clean it up first using the tidytext package.Дisclaimer: this post might not be 100% reproducible because of the amount of data and time it takes to process, but further down I will provide a link to the almost tidy dataset that can be used for tidy text analysis. This dataset was made publicly available during the Federal investigation into EnronвЂ™s accounting/business practices. Text data is pretty fun to work with (at least I think so) and I decided to use the Enron email data for this post (title inspired by the book The Smartest Guys In The Room). In The Office Part 1 I used a fake dataset which was good for what I wanted to demonstrate, but not as interesting as using real data.