{{ $page.post.title }}
{{ mention.node.wmProperty }}
{{ mention.node.content.text }}
{\{ post.title }\}
{% endfor %} ``` You will now get the list of all the posts. ## Conclusion Before I used a very complicated method to achieve the same result, I honestly don't even want to tell anyone about it. This is a much more straight forward, much more elegant solution. It requires much less additional code written. One thing I felt when I was writing this post is a worry that people are reading this need more context. I think I will end writing a "Definitive guide to start a Django project." That way, in the future, I can always refer to the structure that already exists, as opposed to starting fresh every single time. I hope this was helpful. Thanks for reading. --- # Adding a dictionary to a list of dictionaries in Python URL: https://www.rasulkireev.com/list-of-dicts/ Type: tutorial Date: 2020-02-13 Description: I have encountered a problem where I was not able to append a dictionary to a list. In this post, I will explore this issue and show how to bypass that. Tags: webscraping, dictionaries, python, webscraping Content: In my most recent project, I have encountered a problem where I was not able to append a dictionary to a list using `list_data.append(data_dict).` The result was a list of the same dictionaries. In this post, I will explore this issue and show how to bypass that. ## Setup Just so that we are on the same page, this is my setup. I am parsing a page that has an entry for each day of each month. ```January
1
blah
2
blah blah
3
blah blah blah
etc...February
1
blah
2
blah blah
3
blah blah blah
etc...by Rasul
```  You can use this on any website you have built. You can direct users to any link you would like. The cool think about this is that it can also be used as a Feedback button. This is useful because it is always there, users can always see it. Yet, it is not right in your face. You would simply need to change the link and the text. For the link, you can make a TypeForm to collect feedback. ## Final Notes You could combine the two widgets on one page. It will look something like this.  You are a judge on this one. For me, this looks alright. Not great, just alright. Here is the code that will get you those: ```htmlby Rasul
Feedback
``` I hope that is useful. --- # New Project 'Built with Django'! URL: https://www.rasulkireev.com/builtwithdjango/ Type: article Date: 2019-12-12 Description: Launching a new hobby project. I am making a curated list of products built with Django. Tags: webapp, ryankulp, microacquisitions, django Content: Last week, I purchased [Ryan Kulp](https://www.ryanckulp.com/)'s course on buying, improving, and selling microbusinesses [(affiliate link)](https://www.microacquisitions.com/how-to-buy-small-companies/8ayt5). One of the reasons I bought this course was to learn another skill (bseides, data analysis and web developemnt) he help my on a path to become financially independent. The other reason is that I desperately wanted to see how one would purchase a business without having a ton of cash upfront. Finally, I have been following Ryan for a while now, and he is just a cool dude, with [a lot to share](https://www.ryanckulp.com/projects/). This is not a post about this course, though. I will write a short review of the course once I finish it. For now, I want to share a new project I am working on and how I came up with it. The reason I mentioned How to Buy, Grow, and Sell Small Companies is that it helped me to get the idea. In one of the videos, Ryan said that at [Fork Equity](https://www.forkequity.com/)(Ryan's Investment firm), they care about the tech stack that the company uses. They have experience with [Ruby on Rails](https://rubyonrails.org/) and will be able to navigate in the projects once they acquire it quickly. I am a big fan of [Django](https://www.djangoproject.com/) and its [community](https://www.djangoproject.com/community/). If I am ever going to buy a business, I do not want to spend a ton of time learning the tech stack behind it. So I thought it would be cool to have a collection of websites built with Django. Hence my project "[Built with Django](https://builtwithdjango.com/)." This seemed like an excellent idea for several reasons: 1. I get to practice and improve my [Django](/managing-django-with-poetry) & Design skills 2. I get to create a community of like-minded people. More specifically, people who like to create things and like to create them with Django. 3. I have a list of great projects that I could potentially help with or even take under my wing and improve upon. 4. Highlight the brightest mind in the community that I love. 5. In case of success, potentially monetize, by creating a job board and other non-intrusive to users way. The project is now up and running. I published it a week ago. I will not go into more detail about how I was making right now. I will write a post on it later. Here I just want to state that it took me about a week to go from 0 to [MVP](https://en.wikipedia.org/wiki/Minimum_viable_product). The hardest part was the front end, it is my weakness. Otherwise, Django makes it pretty easy to build the backend logic. Love Django. I think this is going to be relatively low maintenance, yet useful project. I hope we get some people sharing their work. Hopefully, this will help grow the awesome community of Django makers. --- # Analyzing FIFA 19 data (III). Machine Learning and Prediction URL: https://www.rasulkireev.com/fifa-machine-learning/ Type: tutorial Date: 2019-12-06 Description: In this post we are going to apply some basic machine learning on our clean dataset. We are going to focus on using Scikit Learn Tags: datascience, machinelearning, fifa, numpy, pandas, python, scikitlearn Content: ## Source code The project source code is in [this](https://github.com/rasulkireev/fifa19-data-analysis) Github repo. You can review the code in this post in 3.1-machine-learning-by-the-book. ##Overview In this post, we go through the process of building a machine learning algorithm. I am not making it from scratch. Scikit-learn has done all the work for us. We need to think about the "business" logic and best practices of using this library. Please note, this is part 3 of our project. Please follow the links to review [Part 1](https://www.rasulkireev.com/fifa-data-cleaning/) and [Part 2](https://www.rasulkireev.com/fifa-data-exploration/), where we talked about Data Cleaning and Exploration, respectively. ## Initial Importing Let's begin. First, we need to import the standard packages and a "clean" dataset (from the previous post). ```python %matplotlib inline # Standard Libraries to import import pandas as pd import matplotlib.pyplot as plt import numpy as np dataset = pd.read_csv('data/processed/clean_dataset.csv', index_col=0) ``` This should get us going. ## Creating a test set. Before we begin any further exploration or analysis, we need to create a test set and set it aside. After you have your data, the first thing you should do is to set aside some test data. This ensures a safe and unbiased process. ### Categorizing Overall Skills into bins. This step helps us get the correct proportions of data from each Overall skill bin when we are splitting the dataset. ```python # Create a column that categorizes Overall Skills level into bins. custom_bins = [i for i in range(45,100,5)] dataset['overall_bins'] = pd.cut(dataset['overall'], bins=custom_bins, labels=[i for i in range(len(custom_bins)-1)]) dataset['overall_bins'].hist() ``` If everything works correctly, you should get a histogram that shows an uneven distribution of players that belongs to each "group."  Let's now split the dataset into the test set and training set. Thanks to Scikit-learn, it is straightforward. We are using the train_test_split function. ```python from sklearn.model_selection import train_test_split strat_train_set, strat_test_set = train_test_split( dataset, test_size=0.2, random_state=42, stratify=dataset["overall_bins"]) ``` Here, we set the test size to 20%, which is the standard proportion to pick. If you have a large data set, with millions of rows, then you can choose a smaller percentage. The random state helps us replicate the result in the future. You can use any integer, but 42 is a geeky standard. > I have procrastinated on this post for a long Time. I think this is just because it seems like there is too much work ahead. I have this subconscious desire to make it perfect, just because all the data science related posts I read seem to be that way. No more. I am at the beginning of my career, and putting this out in the world is way more important than making it perfect. So I try to keep this short, sweet, and simple (the new S3). In the future, when I have more time and experience, you can certainly expect much more detail, depth, and sass in my posts. Now, let's continue. If you want to see the distribution of data along with the bins, you can run the following code. ```python print(strat_test_set["overall_bins"].value_counts()/len(strat_test_set)) print(len(strat_test_set.overall)) ```  ## Prepare the Data for Machine Learning Algorithm ### Separating dependent and independent variables The first thing we are going to do is to separate the dependent and independent variables. Since we are trying to predict the value of the player, it is going to be our dependent variable. Other columns are the independent variables. This is the code to achieve that separation: ```python dataset = strat_train_set.drop("value", axis=1) dataset_labels = strat_train_set["value"].copy() ``` ### Converting categorical variable to numerical We are using Scikit-learn built-in method called OneHotEncoder. This is great and simple to use the feature. _**I have to give Scikit-learn dev team a huge shoutout, they are doing a great job overall.**_ The only categorical variable that we have is 'Position.' I think it is useful to consider player positions because they need different skills and attribute to be better or worse and will, therefore, affect the overall value of the player. ```python # One hot Encoding from sklearn.preprocessing import OneHotEncoder dataset_categorical = dataset[['position']] cat_encoder = OneHotEncoder() dataset_categorical_1hot = cat_encoder.fit_transform(dataset_categorical) ``` ### Scaling your features Scaling is important when it comes to Machine Learning. Having values that vary in the range can through your algorithm off. So keeping everything on the same scale is very important. One way to do this is Scikit Learn's built-in method called StandardScaler. This method applies the [Min-Max feature scaling](https://en.wikipedia.org/wiki/Feature_scaling#Rescaling_(min-max_normalization)) , which essentially yields values from 0 to 1. I use a Pipeline method to apply the scaling. This helps us automate the process in the future by creating a pipeline. This is how the code looks like: ```python from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler num_pipeline = Pipeline([ ('std_scaler', StandardScaler()), ]) ``` ### Creating a Full Pipeline The final step in preparing the data is to create a Full Pipeline that we can use to feed the unprocessed information. In our case, it is the test data that we have prepared in the beginning. To create the full pipeline, once again, you need to use Sciki-learn built-in method called `ColumnTransformer`. It is a relatively new feature. It is very efficient. We are also going to leverage the Pipelines we have created earlier. This is how we do this: ```python from sklearn.compose import ColumnTransformer # Make a dataset with nums only dataset_num = dataset.drop("position", axis=1) num_attribs = list(dataset_num) cat_attribs = ["position"] full_pipeline = ColumnTransformer([ ("num", num_pipeline, num_attribs), ("cat", OneHotEncoder(), cat_attribs), ]) ``` Please note: the third parameter for both functions within ColumnTransfomer are lists of columns that this function applies to. Then we will perform a fit_transform on the dataset: ```python dataset_prepared = full_pipeline.fit_transform(dataset) dataset_prepared.shape ``` `.shape` is here to check the dimensions of the final dataset. This is the result for me `(14516, 59)` ## Applying the Machine Learning Algorithm There is a plethora of different Machine Learning Algorithms that was pre-built by Scikit learn. We are going to use the Random Forest Regressor. It is a robust algorithm. I am not going to go into the details of how it works. There are some excellent resources out there that talk about this in detail. If you followed along and did everything successfully, the following code looks extremely simple to you! ```python from sklearn.ensemble import RandomForestRegressor forest_reg = RandomForestRegressor() forest_reg.fit(dataset_prepared, dataset_labels) ``` That's it. We now need to test the algorithm. One of the simplest ways to do this is to check the Root Mean Squared Error. The following code does not. ```python dataset_predictions = forest_reg.predict(dataset_prepared) forest_mse = mean_squared_error(dataset_labels, dataset_predictions) forest_rmse = np.sqrt(forest_mse) f'RMSE is ${forest_rmse:,.0f}' ``` I get `'RMSE is $374,525`. This means that, on average, the algorithm is only roughly off by $400k, which is not a lot compared to the actual values. We are talking millions here. ### Saving the model Suppose we are happy with our model. Now the right thing to do would be to save it. The easiest way to do this is by using joblib function. It is easily accessible from sklearn.externals. This is how you save the model: ```python from sklearn.externals import joblib joblib.dump(forest_reg, "models/best_model.pkl") ``` Here the forest_reg is the model we want to save and the second parameter `"models/best_model.pkl"` is the path and the name of our new file. You can check the directory to make sure everything saved correctly. ## Evaluating on the test set Now that we have a working algorithm we want to use, it is time to finally test it on the test set to see if it works correctly. We are going to take our test_set and feed it through the pipeline we created earlier. After that, we are going to feed it through the model we have trained. Excuse my "feed-through" language. This is the first thing that comes to mind. ```python final_model = best_model X_test = strat_test_set.drop("value", axis=1) y_test = strat_test_set["value"].copy() X_test_prepared = full_pipeline.transform(X_test) ``` The code above makes the final "prepared file." ```python final_predictions = final_model.predict(X_test_prepared) final_mse = mean_squared_error(y_test, final_predictions) final_rmse = np.sqrt(final_mse) ``` Now we successfully "fed" or dataset thought he model. The last thing to do is to evaluate the error. ```python f'RMSE is ${forest_rmse:,.0f}' ``` `'RMSE is $359,161'` All right! The model performs better on the test set, rather than the training set. This is a little surprising to me. The difference is tiny, so no worries about underfitting. ## Final Thoughts I can't believe this is "over." We have gone through the process of Data Analysis from Data cleaning to Building a Model. There are so many things I have left out, even things I did for this dataset. It is impossible to cover everything in one go. I hope this was at least a little tiny bit useful. There are fantastic books written on these topics. The books that helped me write this post is 'Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow' by Aurelion Geron. ## Final, final thoughts These are my first baby steps in the world of Machine Learning. These are the first baby steps in the world of Data Science blogging or any blogging for that matter. So, please do not be too harsh. You know what, actually forget it. Please, be harsh. I don't have too much free time, and I want to improve as quickly as possible. So, some constructive criticism would be great. This post took me waaay to long to write. Also, I had big blocks of time in between those writing, so I am afraid of small inconsistencies. In the future, I will try to break it up into a much smaller code block. I guess it was a little too ambitious to write a "form start to finish" post as my first post. Well, I am happy I wrote. I am so glad it is now behind me, and I can move on with some other things. I promise to try to be more consistent and more frequent with posts. --- # Just launched my first Web App! URL: https://www.rasulkireev.com/first-web-app/ Type: article Date: 2019-10-29 Description: Finally launched my first Web App. Kushim is my first Django project which was built from scratch. Tags: webapp, django, saas Content: This is a big thing for me. I have been learning Django for several months now (around 4). I built a website with it, thanks to all the great tutorials out there. Building a Web App is a whole different thing though. I have struggled to find time and courage to actually build this thing. It took many different forms and ideas while I was building it. ## Gratitude I want to give a quick shout-out to people/resources that helped me achieve this. I'd like to thank: * my wife for putting up with me during this long and hard process! * the StackOverflow community (especially to [Daniel Roseman](https://stackoverflow.com/users/104349/daniel-roseman?tab=profile) who answers a ton of questions and very quickly I have to say). * [William Vincent](https://wsvincent.com/) for writing a lot of good tutorials. * [IndieHackers Community](https://www.indiehackers.com/) for constant inspiration. * [Coutland Allen](https://www.indiehackers.com/csallen) for building Indiehackers and for the podcast, which is extremely inspirational. * ... This list can drag along pretty far. The point is I am thankful to all the people who helped me with this. Funny enough, none of those people know they have helped me. ## The Process I wanted to build something. Anything, not necessarily a web app. ``` Note: While writing this I feel so shitty for some reason... "Who are you to be writing about this?", "No one cares..." are the thought that comes to my head... ``` I needed some extra money to live off, since my current wage is not enough. This leads me to try a bunch of things. * I tried to build a t-shirt shop with Shopify, that failed. * I tried to build a website with Wordpress to sell some written work. That failed too. * I tried to record myself cooking to make a Youtube channel, I didn't follow through. * I made a business newsletter for Russian Entrepreneurs. No, signups. * I made an online journal so that students can post their work for the world to see. No submissions... I was kind of sad and decided to do something that will at least help me get a better job. The first thing was to make a website. I tried, Wix.com, Jekyll, plain HTML, it was all shiiiiiiiit. Really, really bad... **This is turning into my life story, which it is not supposed to be. I will stop now and will continue this story sometime in the future, promise.** Long story short, I decided to learn Django and after making the website I was happy about I decided to build some kind of SaaS product that people would pay for. One of the ideas was inspired by Derek Sivers. More specifically, by hist post on [Journaling](https://sivers.org/dj). There he talked about the benefits of journaling. > If you care about your thoughts, keep them. One of the ideas was to also have "Thought On" Journals, for all the different thought you might have. For example, Derek has Journals about Airports, Sex, Travel and a bunch of others. I thought that was a great idea for a web app. Allow people to create journals for any thought that they have. That's it. That was the premise of the app and I have now built it. Now you can do that. Building it was not very hard with the knowledge I gathered in roughly 3 months of building my personal website. The hardest thing was to actually start building it, finding time to continue building it and to have the balls to follow through and finish building it. Somehow, I managed to do all that and am still pleasantly surprised that I did. Just as a remark I want to add that it wasn't super easy all the time, I ran into errors all the time. All. The. Time. You just need to be an adult and seek help. StackOverflow is a great place to ask for help. ## Pricing and Next Steps I thought that this is not enough to ask people for money and so I decided to make it free until I build the next feature. The next feature will be akin to a Personal CRM like [MonicaHQ](https://www.monicahq.com/). I think the price I will be asking for is around 4.99 a month. I won't be expecting much excitement or registration at that price point. If someone signs up I will be super happy, otherwise, I will be happy just using it myself. ## Final Thought Just because this is the first post about me building something, I wanted to tell a lot more, but it is very easy to go down the rabbit hole of telling you each small detail about my thought process and all the vents that happened to me before I actually published. Life is a hard and complicated thing. It is very hard to isolate specific events that lead to other events. Nothing happens in a vacuum. Hopefully, if I continue to work and write about it, I won't have to talk about the details so much and a lot of things will start to become self-evident. --- # PyGotham 2019: Day 2 URL: https://www.rasulkireev.com/pygotham-day-two/ Type: article Date: 2019-10-15 Description: Second Day of my first conference ever. I tried to make good notes during the talks I attended. Tags: conference, networking, pygotham, talks Content: Day 2 of the PyGotham conference was on Saturday. I expected there to be way more people, but surprisingly, the opposite has happened. Day 2 count was twice as small, or at least that is how it felt. Below are the note from Day 2. You can see Day 1 notes [here](pygotham-2019-day-one) _Please note: I took some pictures, but am not able to upload that as of this second. As soon as I can, I will updateCreated the post._ _Please note: I took some pictures, but am not able to upload that as of this second. As soon as I can, I will updateCreated the post._ *** # 1000x faster data manipulation: vectorizing with Pandas and Numpy ## Nathan Cheever @AdvancedMD Ds * For loop vs vectorization * Vectorization working with array or series without for loops, rather all at once. * You are probably already doing that to some degree * To perform conditional vectorization do: **numpy.where()** * Numpy.vectorize() * Numpy.select() for multiple ifs / elifs * Pd.shift() to create anoer plumn with values shofted * talked about Dask * Kubernetes shirt *** # Using Dash to Create Interactive Web Apps for Non Technical Audience ## [Joseph Willi](https://www.linkedin.com/in/joseph-willi-38b7883b/), @[underwriters labarotory](https://www.ul.com/) ### AZV Dashboard (Arizona Vortex) * Great company, mission oriented * Their job is to fire staff and gather data * Somehow connected physical devices such that the plotly days gathers data from the device and inputs it to the dashboard *** # Distributed Machine Learning with Python ## [Brad Miro](http://https://www.linkedin.com/in/brad-miro/) @google * Latest ml updateCreateds,: Gpu, Tensorflow unit * Latest ml updateCreateds,: Gpu, Tensorflow unit * Kubeflow, marries Tensorflow and Kubernetes *** # Pull Requests: Merging good practices into your project ## [Luca Bezerra](mailto:lucabezerra@gmail.com) - vinta * Code review , even if unwanted adds greatly to the project * Best practices: * Pull request template.md on github * Status checks prevent branches merging without checks * Contributing .md for collaboration * Enforce approval/merge rules : codeowners.nd * Using **gitflow** makes sure your branches are always up to dateCreated * Using **gitflow** makes sure your branches are always up to dateCreated * Branches are cheap and great flexibility * Keep pr size as small as possible * More clear commit messages (talk about what and how ), * Use linters when possible * Tips & tools * Refined GitHub * Deploy Previews (Netlify) * Check Vinta checklist *** # Jane Doe will help you improve your project ## Rebecca Sarai - Vinta * Netflix prizes - challenge * Talk is about anonimization * Psedonymization - is personal data , but you hide the data in a particular manner , includes * Tokenization, etc. * Anonimization no data that considered personal and can be sued to identify a person * Static * Dynamic * Synthetic *** # The Cat's Alive! Add Some Weirdness To Your Code With Quantum Computing ## Jorn Mossel, Marianne Hoogeveen * Classical computer works with 0 and 1 * Qubit can be a bit of both (circular form) *** # Oh, the Humanities! Interdisciplinary Thinking in Python ## [Lisa Tagliaferri](https://lisatagliaferri.org) - wrote tutorials for Digital Ocean * Research on how women men are described in certain ages. * Text from Gutenberg projects were analyzed * Very interesting * Text analysis might not be that scary after all. It never was , bit now it is certainly interesting enough to give it a try * Gender novels project * [Caterina.io](https://Caterina.io) * [Hacktober fest](https://hacktoberfest.digitalocean.com/) *** # Serverless Deep Learning with Python ## [Alizishaan Khatri](https://www.linkedin.com/in/alizishaan-khatri-32a20637/) @Kony Machine Learning Engineer * Kony builds applications for banks * Unfortunately I do not follow at all *** # Airflow in Practice: How I Learned to Stop Worrying and Love DAGs ## [Sarah Schattschneider](https://www.linkedin.com/in/sarahschattschneider/) - software engineer @[blue apron](https://www.blueapron.com/) - hiring * One of advantages is the more advanced alerting system * Good cli and UI * Open Sourced by Airbnb --- # PyGotham 2019: Day 1 URL: https://www.rasulkireev.com/pygotham-day-one/ Type: article Date: 2019-10-05 Description: First Day of my first conference. I tried to make good notes during the talks I attended. Tags: conference, networking, pygotham, talks Content: This is a first conference I have ever attended. If I had to describe it in one word, it would be 'great'. Here is why: * I've met a lot of people with similar interest personality types, which is hard to do in regular setting. * I've made a number of "useful" contacts that might help me find a good job. * I've seen a lot of interesting presentations that have sparked my interest in programming (in Python) even more. During the event I took notes with [Joplin](https://joplinapp.org/) , which is a great open-source note-taking app. I strongly recommended to everyone. **Not a sponsor**. The general structure of the notes you are about to read is the same: # Name of the talk ## Author * Notes are going to be bullet points. `If i have any notes above the bullet point above, I will leave them in this grey {code} block` * I will try to leave find as links as possible, so you don't have to google everything. So, without further ado, here are the notes I've taken during the first day of the event. *** # Python use spectrum ## Kojo Idrissa * More speakers always needed. `This makes me want to make a talk to present at one of the conferences, seems totally doable and great for portfolio.` * Volunteers always needed `Certainly feel I should have been volunteering at this event. Will try to consider volunteering much more often` * [Defna.org](http://Defna.org) - Django organization *** # Eita! Why internalization and localization matter ## Nicolle Cysneiros [@Labcodes](https://labcodes.com.br/) - (Web Design Agency whos tech stack is python, django and js) ### Differences in formats across countries * Date Formats (mm/dd/yyy vs. dd/mm/yyyy, etc) * Number Formats (123,123 vs 123.123) * Currency, [Bidirectional Text](https://en.wikipedia.org/wiki/Bidirectional_Text) & [Unicode Characters](https://www.rapidtables.com/code/text/unicode-characters.html) differ from country to country. ### Python library to deal with that * GNU [`gettext`](https://www.gnu.org/software/gettext/) * This piece of software essentially is helping you to create a 'list' of text blocks that will need to be translated. * [`locale`](https://docs.python.org/3.7/library/locale.html) module. * This module helps change the formats of the above mentioned differences. * Apparently Django makes it super easy. *** # Can you keep a secret? ## [Aaron Bassett](https://www.gitshowcase.com/aaronbassett) @[Nexmo](https://www.nexmo.com/) * [Git-Secret](https://git-secret.io/) * Uses [pgp](https://en.wikipedia.org/wiki/Pretty_Good_Privacy) *** # Absolutely Awesome Automated APIs ## [Timothy Allen](https://www.linkedin.com/in/flipperpa/) - @[WRDS](https://wrds-web.wharton.upenn.edu/wrds/index.cfm?) (uses Django) ### -@flipperpa -djangonaut -hockey_fan_flyers -guitarist -loves_icecream * WRDS developed and open sourced [Automagic-Rest](https://pypi.org/project/automagic-rest/) * [Drf renderer xlsx](https://github.com/wharton/drf-renderer-xlsx) - open source library to convert API endpoint to excel file *** # Recreating "The Clock" with Machine Learning and Web Scraping ## [Kirk Kaiser](https://www.linkedin.com/in/k-p-kaiser-94500124/) @[datadog](https://www.datadoghq.com/product/) * [Tate modern - the clock](https://www.tate.org.uk/whats-on/tate-modern/exhibition/christian-marclay-clock) * To start, simplify: try kubernetes locally - Ubuntu + Nvidia, [ngc.nvidia.com](https://ngc.nvidia.com) * [Dab & t-pose controlling light](https://github.com/burningion/dab-and-tpose-controlled-lights) * [Jetson nano gpu accelerated machine 99$](https://amzn.to/2oyUSsS) * Datadog is hiring *** # To comment or not? A data-driven look at attitudes toward code comments ## [Veronica Hanus](https://www.linkedin.com/in/veronicahanus/) * Funny, good presenter * The general consensus is that comments are good and you should leave them *** # Learning with Limited Labeled Data ## Shioulin Sam @cloudera * Very clear and great presenter * **Active learning** has been here for a while, but only now got integrated with ... * Random sample data points to label (good, but not optimal) * [Margin Sampling](https://link.springer.com/chapter/10.1007/978-3-662-45646-0_13) * [activelearner.fastforwardlabs.com](https://activelearner.fastforwardlabs.com) * [Twiml and Ai podcast](https://twimlai.com/) *** # Introduction to Generative Adversarial Networks (GANs): hands-on to making new data (and some pretty pictures) ## [Nabeel Seedat](http://nabeelseedat.com/) * [Dirpfakes](https://twitter.com/realderpfakes?lang=en) * [Deep Convolutional GAN](https://medium.com/@jonathan_hui/gan-dcgan-deep-convolutional-generative-adversarial-networks-df855c438f) * [Fashion Mnist](https://github.com/zalandoresearch/fashion-mnist) * [Gan Zoo](https://github.com/hindupuravinash/the-gan-zoo) *** # What physics can teach us about learning ## [Marianne Hoogeveen](https://www.linkedin.com/in/marianne-hoogeveen/) * Neutral network can approximate any smooth function - [Universal approximation function](https://en.wikipedia.org/wiki/Universal_approximation_theorem) * Scales matter when it comes to identifying patterns (neural networks) - [renormalization group](https://en.wikipedia.org/wiki/Renormalization_group) i.e. compression *** # Django + ElasticSearch without invalidation logic ## [Flávio Juvenal](https://twitter.com/flaviojuvenal?lang=en) @[vinta](https://www.vinta.com.br/) * Most popular search engine * Why use search engine * Tokenization * Fuzzy similarity * Cleaning * Elastic search built ins * Less than million rows of Data can be searched with postgres built-in capabilities (check Django postgres search ) if more than 1m than implement elastic search * [Haystack library](Haystack library) for search in Django * [Zombodb](https://github.com/zombodb/zombodb) works with postgres (via access method API) (Django-zombodb) * will Soon work with postgres 11 *** # Static Typing in Python ## [Dustin Ingram](https://dustingram.com/) @Google @di_codes * type errors happen * When Python expects a list received a string and converts that to a list , ugly list * Python originally is dynamically typed, but can also be static. Which is a great strength * Pep 3107 fucntions * Def Foo() <- max(2,9) * Foo.__annotations * [Mypy](http://www.mypy-lang.org/) from PhD introduced on pycon us 2013 * Experimental variant of Python that is statically typed * After talking to Guido decided to implement to pure Python * pep 483 theory of type hints * Optional typing (+gradual typing) * Variable annotation * Check data science from scratch again, it has some static type instructions * Pep 484 type hints * `Pip install mypy` * When not to use : never * When to use : as much as possible (Start early) * When code is confusing * When for public consumption * Before big migrations and refactoring * Start early * Note : not a replacement for unit test *** # Convincing an entire engineering org to use (and like) mypy ## [Annie Cook](https://www.linkedin.com/in/annielcook/) @[nylas](https://www.nylas.com/) - hiring * [Mypy](http://www.mypy-lang.org/) is a static typechecker , type of documention (self documenting ) ```python #pep 484type hints Def Foo(type: str) -> str Return ``` * With mypy linters work better * Captures a large category of errors * Union is a multiple type * _learn x in y minutes_ . Saw some guy in the talk use this website, seemed interesting *** # ministry of silly talks ## [in-toto](https://www.In-toto.io) - securing the whole software supply chain ### nyu , njit, Santiago Torres Aria's, Hammad Afzali, etc. * Not rely clear, probably hard to implement * Sounds scary where people can attack software, need to check if that actually happens ## Test your failures with xfail ### [Paul Ganssle](https://ganssle.io/) * Decorate a test that you expect to fail with `@pytest.mark.xtest` ## Experimental features in [Scikit-Learn](https://scikit-learn.org/) ### [Thomas J Fan](https://www.linkedin.com/in/thomasjpfan/) - scikit learn core devoler * Deprecations? * Use columntrasformer to transformer some of the dataframe ## Why your tech company from recruit from philosophy department ### [Katherine Hartling](http://www.katbelle.com/) - @KatOnceSaid * [Jackpocket.com](https://jackpocket.com/) * [Plato's concept of classes](https://oregonstate.edu/instruct/phl201/modules/Philosophers/Plato/plato_on_women_in_the_ideal_state_part_I_the_context.html) ## Greed an homage ### Julian, @[julian](https://github.com/Julian) - GitHub, @[julianwastaken](https://twitter.com/julianwastaken?lang=en) on Twitter * Algortim of change in supermarket, greedy algorithm * when does it work * When you can take Independent choices within each step * [Matroids](https://en.wikipedia.org/wiki/Matroid) ## Change just one thing ### Timothy Allen , same guy as WRDS talk in API * Dealed with alcohol addiction problem! * Ask help, people are willing to help, especially the Python and Django communities ## Structured logging in Python ### [Jonathan Meier](https://jonathanmeier.io/) software engineer @[openslate](https://www.openslate.com/) * Structured logging is writing consistent logs that are easy for computers to read (JSON) * [Python-json-logger](https://github.com/madzak/python-json-logger) or structlog ## Restructuring Data in Python ### Mahmoud Hashemi @ sedimental.org * Flat is good vs namespaces are good. Tim Peters - Pep20 * Nested data is bad. Tim Peters * [Glom library](https://glom.readthedocs.io/en/latest/) - Python data nester * Good presenter. ## Ministry of silly runtime: Vintage Python on cloud run ### Dustin Ingram @Google @di_codes * Run stateless http container * Docker images as a service * Installed old Python (1.0.1) on cloud run * Good presenter --- # Analyzing FIFA 19 data (II). Data Exploration and Visualization URL: https://www.rasulkireev.com/fifa-data-exploration/ Type: tutorial Date: 2019-09-26 Description: My thoughts on the Open Source community and the culture behind it. Tags: opensource, plotly, pandas Content: You can find all the code to this project [here](https://github.com/rasulkireev/fifa19-data-analysis). You can review the code in this post in [2.0-plotting-and-exploration.ipynb](https://github.com/rasulkireev/fifa19-data-analysis/blob/master/2.0-plotting-and-exploration.ipynb). In this post I will go through the process of data exploration. In [part 1](https://www.rasulkireev.com/fifa-data-cleaning) we already did some "exploration and are now much more familiar with the dataset. In this part I will mostly focus on visualizations. To be more specific, I will focus on visalizing our data set with a python library - ["Plotly"](https://plot.ly/python/). Before we continue I just wanted to quickly explain why I will be using Plotly, instead of matplotlib or seaborn. 1. Plotly graphs and charts look really good, even without a lot of customization. 2. There are some useful features taht will allow me to explore the dataset (for example, I can zoom in on the chart interactively). 3. We can later make a ['Dash'](https://plot.ly/dash/) dashboard, which uses plotly for all its graphs. ## Installation If you are using anaconda, you have it pre installed. If you are using your own virtual environment then install simply with ```python pip install plotly ``` I strongly suggest you use [jupyter notebook](https://jupyter.org/) for all the analysis and exploration. Again if you are using Anaconda environemnt, you have it pre-installed and simply need to launch jupyter notebooks from the Anaconda Navigator, which is a programm you can run on your computer. If you decided not to use Anaconda than you can easily install jupyter notebooks simply following the official [installation guide](https://jupyter.org/install). Plotly should be working fine with Jupyter Notebooks, right out of the box, but if you are running into some errors, try installing ipywidgets using pip, and generaly going throught the plotly jupiter lab [integration guide](https://plot.ly/python/getting-started/). ## Importing packages Since we are not working with matplotlib or seaborn, we only need to import a couple packages: ``` python import pandas as pd import plotly.graph_objects as go import plotly.express as px ``` * we need pandas beacuse we are wroking with a dataframe * plotly.express is good for quick plots that don't require any customization * plotly.graph_objects allow for full customization Finally, before we begin let's import the dataset we made in the previous post. Please make sure you are wokring in the same directory as you csv file. ```python combined_columns_data = pd.read_csv('reduced_clean_data.csv', index_col=0) ``` ## Correlations using `scatter matrix` ### Basic Player Info (Height, Weight, Overall, Value, Wage) One of the first things we want to look at is something called a scatter matrix. This chart will allow us to compare a bunch of variables (columns) to each other to see if there are any interesting relationships going on in our data. With plotly express it is pretty simple to do. ``` python fig = px.scatter_matrix(combined_columns_data, dimensions=['height','weight','age','value','overall','wage'], color="position") fig.show() ``` As you can see in the code above, we called plotly express graph called `scatter_matrix`, told it what data we want to use (`combined_columns_data` dataframe), specified which variables we are interested in and I also decided to add a color distinction for each positionm so that we could see if there are any clusters. Here is the output:  Nice. Now let's see if there is anything interesting to note. Here are my observtions (if you see something I missed, please [let me know](mailto:kireevr1996@gmail.com)): 1. _(column 2, row 3)_ The value of the player is roughly the same when the Overall Rating is between 0-80. Once the rating passes the 80 mark, the value starts to climb up quickly. 2. _(column 2, row 4)_ Same thing happens to wage. 3. _(columns 5 & 6, row 4)_ These resemble normal distribution plots. This makes sense, since height and weight tend to be normally distributed across any population. what is interesting is that Most valuable or highly paid players tend to be closer to the mean. 4. Finally, as we can expect there is a correlation betweem height and weight. The tallker the playeer, the more he weighs. Let's continue. ### Detailed PLayer Statistics - (Speed, Passing, Physical, Control, Defending, Skill, Shooting) Now let's look at the variables we prepared in the previous part. I want to see id there is any correlation between skills. Currently I expect that there is in fact strong correlation between most skills like, control, shooting, passing, generally because better player will have all detailed stats higher than p[layers that are worse overall. However, specific skills like Defending and Goalkeeping will not be correlated with the rest, since they are very specific. For example, any Goalkeeper will have much higher "Goalkeeping" rating than any other position, conversly he will have much lower Speed or Control compared to other positions. As in the previous example we first create a graph using the express tool. Then we ask to display it with `.show()`: ```python fig_two = px.scatter_matrix(combined_columns_data, dimensions=['speed','passing','physical','control','defending','skill', 'shooting'], color="position") fig_two.show() ``` Here is the result:  As expected, Goalkeepers (the purple color) are a separate group in all GK graphs. Additionally, LB, RB, and CB all form a cluster on the "Defending" graphs. These detailed variables will be useful to differentiate between the group pf people whos overall is around 70 (mean), which is the most dense area. ## Visualizing key metrics The main features of our data is the value and skill. Let's see how one affects the other a little closer. Again, plotly let's us do that pretty simply: ```python fig = px.scatter(combined_columns_data, y="value", x="overall", color="age") fig.show() ```  Interestingly, the relationship is exponential and we will probably have to use a polynomial model in the Machine Learning part of the project. > Note: I should have separated the dataset into test set and training set before starting the exploration. You should too. We will see how to split your data in the next part. In the chart above we are displaying 3 metrics: Player Value, Overall Skill Level and Age (color). This is great, we can see that the bottom corner is more of a yellow color which tells us that player of the same level but different skill level have different value, with older being less value, which makes sense. We can try to add another metric: size, by adding a `size` method. ```python fig = px.scatter(combined_columns_data, y="value", x="overall", color="age", size="skill moves") fig.show() ``` This does not add any value to the graph as it becomes overloaded with information. Keeping charts simple and easy to read is very important. ## Bonus: Working with separate "Positions" Now that we looked at the dataset from a high level, let's take a closer look. My experience with FIFA tells me that each position has its own strengths and weaknesses when it comes to skills, which is why I decided to break down our data into chunks for each position (GK, CB, ST, etc.). One way to do that would be to import all the different csv files we made and make a dataframe for each of them. This is too much writing and too costly for the computer, memore-wise. Instead I will make a function that will create a dataframe on the fly, depending on the position we want. Here are the steps I have in mind: 1. Create a list of unique names. 2. Create a DataFrame dictionary to store key:value pair of future dataframes, i.e. {"GK":pd.Dataframe}) 3. Iterate over the keys in the dictionary to create dataframes with a name DataFrameDict[key], i.e. DataFrameDict["GK"] ```python #create unique list of names UniqueNames = combined_columns_data.position.unique() #create a data frame dictionary to store your data frames DataFrameDict = {elem : pd.DataFrame for elem in UniqueNames} for key in DataFrameDict.keys(): DataFrameDict[key] = combined_columns_data[:][combined_columns_data.position == key] ``` With this in place you can perform all of the above visualizations, but only for a position you want. For example, instead of doing ```python fig = px.scatter(combined_columns_data, y="value", x="overall", color="age") fig.show() ``` you will now do ```python fig = px.scatter(DataFrameDict["GK"], y="value", x="overall", color="age") fig.show() ``` This will output a graph with only Goalkeepers as data points. ## Conclusion In this post we explored the data exploration with Plotly. Plotly is great and powerful tool making some of visualizations very quick and easy. Let me be clear, I do not discard other tools like Matplotlib, the father/mother of all visualizations in Python, Bokeh and others. They do have some advantageous and disadvantageous. Plotly takes care of a lot of functionality for you which might be useful for you data exploration project. We only looked at plotly express package, which is extremely easy to use, but doesn't offer much customization. If you do require some more formatting [plotly go](https://plot.ly/python/) should be used. In the next post we will look into Scikit-learn and will build a model to predict the value of the player. If you have any questions or comments, please email me at [me@rasulkireev.com](mailto:me@rasulkireev.com). --- # Setting up Python virtual environment (for a Django Project) URL: https://www.rasulkireev.com/django-venv/ Type: tutorial Date: 2019-08-29 Description: How to set up a virtual environment for your Django Project, or any other Python project, for that matter. Tags: virtualenvironment, python, django, project, venv Content: For this tutorial I will assume you have a little experience working with the command line tools like "Terminal". If you don't I promise to make a post or two about that, meanwhile you can check those two out: * [Mac](https://www.dummies.com/computers/macs/mac-operating-systems/how-to-use-basic-unix-commands-to-work-in-terminal-on-your-mac/) * [Windows](https://www.cs.princeton.edu/courses/archive/spr05/cos126/cmd-prompt.html) ## Setting up the environment _The first steps is here for people who don't yet have a folder where they keep all their projects._ ### 1. In the Terminal or Command Prompt (cmd) create a directory (folder) where you will keep your projects and websites. ``` mkdir sites cd sites ``` `mkdir` will create a new directory and `cd` will move into into that directory. You can call your folder whatever you want. ### 2. Creating a project folder Once in the `sites` directory you will need run `mkdir` again to make a folder specifically for your project ``` mkdir {projectname} cd {projectname} ``` ### 3. Setting up the environment Before we initiate a django project we need to setup a python virtual environment. This will make sure that whatever machine we are working with (Mac vs. Windows, for example) the website will work by setting up the same packages and libraries. _Generally, having a virtual environment for each one of your projects, whether Web Dev, Data Science or General Programming, is now a standard. For one, this allows other people to help you with you projects and generally makes eveything easier in the future._ For this step you need to have Python installed. I am not going to go over the installation process and will assume you have a working version of Python 3.6 or above. To create a virtual environment python has a built-in package `venv`. ```python # If you have multiple python versions on you computer python3 -m venv venv # If you have python 3.6+ installed only python -m venv venv ``` Here the first `venv` refers to the program we are running. The second `venv` is referring to the name I gave my virtual environment. So, instead of a second `venv` you can type in whatever you want. I find the word venv pretty easy to rememeber and to type up, so that's how I usually call my virtual environments. Let the prgoram run. Congrats you now have created a virtual environment! You can check that everything worked by checking the contents of the folder. You can do that by navigating in `Finder` or `Windows Explorer` or even better just by typing `ls` to the command prompt, this will list out all the directories in your project folder. All there is left to do is to activate the environment to start the working process. On Mac to activate and deactivate the virtual environment you would use the following command: ``` # to activate source venv/bin/activate # to deactivate deactivate ``` On Windows: ``` # to activate .\venv\scripts\activate.bat # to deactivate deactivate ``` When your virtual environemnt is active you will see the name of the virtual environment within bracket to the left of the command line. Like so:  ### 4. Create a Django project Finally, we can create the project. Once you make sure your venv is activated, all you have to do is: ```python # to install django within your virtual environment pip install django # to actually create a project django-admin startproject {name of your peroject} . ``` Note the **dot** in the end of the command. You actually need to add it too. This will make sure that the project is created within the directory you have already created. One last step: ```python python manage.py runserver ``` This will start server with your Django project, If you go to [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser, you will see a welcome to Django screen. **Congrats!** You can now develop your site with Django freely, which is a whole another topic of discussion. If you have any questions please let me know at one of the social links on the top left. --- # First Javascript Experience URL: https://www.rasulkireev.com/first-javascript-experience/ Type: tutorial Date: 2019-08-28 Description: Sharing my first epxerience with Javascript. Had to add some action to my new web app. Tags: webapp, javascript, html Content: I have been reading a [JavaScript book](https://amzn.to/2KPIsoQ) for a couple of months now... I didn't learn much, unfortunately. I know this is becasue all reading need to be supplemented with some sorts of exercise, preferrably real world projects. Luckily, I have an idea for a project that will require some JavaSript. So, instead of reading a book, I will start by formulating an idea and Googling it (or rather StackOverflowing it). In this post I will go over my first (baby) steps in JavaScript. As far as I understand you can either write some JavaScrip (JS) directly in the html code, like so (the example is from [W3Schoools](https://www.w3schools.com/js/default.asp): ```