How do you deal with near duplicate data, or join two datasets with some errors? For example when a book is added to Open Library it's easy for accidental duplicates to occur, and there are many in practice. There are often small differences between duplicates, such as abbreviating author's names, subtitles, omitting the words like "the" and "a". Another related problem is joining to another database; for example book and author names extracted from text.

I'm working on a project to extract books from Hacker News. Most HackerNews posts aren't about books, and it would be extremely tedious to manually annotate examples when most of them are negative. Instead I used different heuristics to determine whether a book contains a title, that can then be used for weak labelling. My main takeaway is that zero shot classification seems like a great starting point for building a classifier.

SentenceTransformers provides a convenient interface for creating embeddings of text (and images) in PyTorch, which can be used for neural retrieval and ranking. But what if you want to integrate a model trained in SentenceTransformers with other existing models in Tensorflow? The best solution would be to rewrite the training in Tensorflow, but if you’ve already spent a lot of time training a model you may want to import it into Tensorflow.

I'm currently working on a project to extract books from Hacker News to help find interesting books. Having extracted extracted ASINs from Hacker News posts and linked them to Open Library records I have a basic proof of concept. Now I want to be able to display this information in some webpages to help people find the books. I ended up building a minimal prototype by manually curating some examples; this let me focus on the design rather than on the technical aspects.

The book Human-in-the Loop Machine Learning by Randall Munro has a code example of finding hazards in food safety reports. Here's the description of the problem: Food Safety professionals want to collect data from incident reports about where pathogens or foreign objects have been detected in food. “I want to maintain a complete record of all recorded food safety incidents in the EU” “I want to track when different food safety incidents might have come from the same source” “I want to send warnings to specific countries when there are likely to be food safety incidents that have not yet been detected or reported” The interface has fields for "Hazard", "Food", "Origin", and "Destination" along with a short extract of text from a food report.

I've been looking at Open Library as a knowledge base for books. After extracting ASINs from Hacker News posts, I want to link them to Open Library records. For books an ASIN is the same as its ISBN-10, which creates a linkage point with Open Library. From my previous Open Library Exploration about 20% of editions in Open Library has an ISBN-13, but not an ISBN-10 (and 64% have one of either).

The book Human-in-the Loop Machine Learning by Randall Munro has a code example of annotating headlines for a data analyst. First you choose a topic name and then can annotate examples. For example I chose "sports results" intending to label headlines containing the result of a sport contest (and not other kinds, e.g. political contests). It wasn't totally obvious how to annotate examples at first; I had to click in the box with the example headline.

Most machine learning models are guided by human examples, but most machine learning texts and courses focus only on the algorithms. You can often get state-of-the-art results with good data and simple algorithms, but you rarely get state-of-the-art results with the best algorithm build on bad data. Robert (Munro) Monarch, Human-in-the Loop Machine Learning Human-in-the Loop Machine Learning by Robert (Munro) Monarch is an excellent book on annotating data for Deep Learning practitioners in industry.

I'm working on a project to extract books from Hacker News, and want to link the books to records from Open Library. I've already looked at the process of adding a book to Open Library and loading a data export into sqlite. Now I really want to look through the data and see what's inside. I do this through two different perspectives; summarising the metadata and looking at some specific records.

I've been looking at Open Library as a knowledge base for books. A lot of the data here is manually uploaded by people, and the user interface has a big impact on the data they enter. To understand the data better I wanted to actually go through the process of adding a book, Overcoming Floccinaucinihilipilification by Jon Manning, and this article documents the process. Overall Open Library prioritises making it easy for people to add data, and then has facilities to edit it.

I'm working on a project to extract books from Hacker News, and want to link the books to records from Open Library. The Open Library data dumps are several gigabytes of compressed TSV, so are too big to fit in memory on a standard machine. The Libraries Hacked repository imports it into a PostgreSQL database, but I don't want to set up a Postgres database for an analysis job if I can avoid it.

I'm working on a project to extract books from Hacker News. Once I have extracted book titles (e.g. with NER or Question Answering) I need a way to disambiguate them to an entity, and potentially link them to other information. The Internet Archive's Open Library looks like a very good way to do that. Book titles can be ambiguous and so we need some way to link it to a unique entity.

I've been reading Robert Monarch's Human-in-the-Loop Machine Learning and the second chapter has a great practical example of human-in-the-loop machine learning; identifying whether a news headline is about a disaster. A lot of the Data Science books, courses, and competitions take for granted that you've got a well defined machine learning metric to optimise, and labelled dataset. In practice you often have a practical objective, and have to define the metric and collect the input features and labels.

I'm working on a project to extract books from Hacker News. I've been thinking about ways to bootstrap this process such as transfer learning, weak labelling, and active learning. I was reading Robert Monarch's excellent book Human-in-the-Loop Machine Learning where he gives good reasons as to why you should start with a separate random dataset for evaluation; you want something that is representative of the real distribution, and any use of a model inherits the biases of that model.

I'm working on a project to extract books from Hacker News. I've previously found book recommendations for Ask HN Books, and have used the Work of Art named entity from Ontonotes to detect the titles. Another approach is to use extractive question answering as a sort of zero-shot NER. This works amazingly well, at least providing that there is an actual book title there. The code is simple using Transformers high level Question Answering Pipeline.

I'm working on a project to extract books from Hacker News. Most HackerNews posts aren't about books, so we need some heuristics to get posts somewhat likely about books. I've already used ASINs to extract book links to Amazon; another approach is like MapFilterFold to use Ask HN threads about books. Ask HN is a kind of post on Hacker News that allows asking questions to the community. We can't identify them in the dataset but they typically start with "Ask HN" in the title.

I'm currently working on a project to extract books from Hacker News. After exporting all 2021 posts from the Google Bigquery dataset in a Kaggle Notebook and doing an exploratory data analysis I'm looking for methods to extract books. One way to extract books is using Amazon links. People often refer to a book with a link to Amazon, and each Amazon product has a 10 character ASIN (Amazon Standard Identification Number); for books this is the same as its ISBN-10.

A mystery! A riddle! A puzzle! A quest! This was the moment that Ada loved best. Ada Twist, Scientist, Andrea Beaty This is an exploration of 2021 Hacker News posts as a precursor to building a books dataset. The data was sourced from the Google Bigquery public dataset bigquery-public-data.hacker_news.full using a Kaggle notebook. SELECT * FROM `bigquery-public-data.hacker_news.full` where '2021-01-01' <= timestamp and timestamp < '2022-01-01' I want to get a basic understanding of what’s in the dataset before doing any data mining.

I'm starting a month long project to extract book titles from Hacker News using Named Entity Recognition. I've been thinking lately about how Data Science can learn from the practices that have emerged in software development, and wanted to find good books on the subject. A lot of the ones I'd read, such as Feathers' Working Effectively with Legacy Code had come out of Hacker News. However this is a hard thing to search with using traditional search techniques.

When using NLP in HTML it's useful to extract the HTML tags which change the meaning of the text. I've already shown how to source map the text of HTML using Python's inbuild HTMLParser. We'll now adapt this to the task of getting the tags, which could be considered as a sort of annotation on the text. I'm actually surprised there's not good tooling for doing this in Python. It seems like the fast HTML parsers like lxml, html5-parser and Selectolax don't make the source positions accessible through their API.

I started trying to write a grammar for generating HTML and quickly got stumped by how to represent HTML comments. From the whatwg specification: Comments must have the following format: The string "<!--". Optionally, text, with the additional restriction that the text must not start with the string ">", nor start with the string "->", nor contain the strings "<!--", "-->", or "--!>", nor end with the string "<!

Sometimes I want to extract text from HTML for processing, but I don't want to lose the context. This is useful in NLP with HTML because sometimes the context (is it emphasised, or in a header or list item) may be relevant for capturing information. The common approaches of converting HTML to text are lossy; you strip away the HTML and lose the context. Instead we're going to do create a source mapping using the Python HTML Parser, for each character of the output text we can find the corresponding character of the HTML that generated it.

I spend most of my time with software barefly aware of the towers of abstraction below. When I click a link in my web browser it sends a HTTP request down through to the kernel which sends the appropriate bits to a network card that are sent to my router, out through to my ISP that does more work to resolve the domain to an IP address to route the query to.

We’re going to build up a simple Deep Learning API inspired by fastai on Fashion MNIST from scratch. Humans can only fit so many things in their head at once (somewhere between 3 and 7); trying to grasp all the details of the training loop at once is difficult, especially as we add more features to it. The right abstractions can make this much easier by only having to think about what we’re changing in the interface.

Chapter 4 of the fastai book covers how to build a Neural Network for distinguishing 3s and 7s on MNIST from scratch. We’re going to do a similar thing but instead of building the neural network from the ground up we’re going to use fastai’s layered API to build it top down. We’ll start with the high level API to train a dense neural network in a few lines. Then we’ll redo the problem going deeper and deeper into the API.

Going from batch processing to near-real time applications is a big conceptual leap for data scientists. Data scientists are often familiar with big SQL analytics databases and can run a batch process weekly or daily. However to get good performance in some applications requires aggregating information that changes more quickly than batch processes can handle. There is normally some lag time between an event being processed and being ingested into an analytics database (and this lag time can vary across the data) which limits the batch approach.

I built Emotion Escape an adventure game with the twist that you navigate using facial expressions. Specifically you upload a photo of a face that is one of happy, sad, angry, fearful or disgusted. The emotion expressed is detected using a machine learned image recognition model, and the story changes depending on your emotion. I built the game in a few hours at no cost, and all the resources are publicly available.

Common Crawl releases columnar indexes of their web crawls in the Apache Parquet file format. This can be efficiently queried in a distributed manner using Amazon Athena or Spark, and the Common Crawl team have released a number of examples. I wanted to see if I could use Python to directly read these Parquet files to find crawls of particular websites. The default way to do this is using their capture index, but this is often under high load so can fail and will reject too many simultaneous requests.

I trained a Transformer model to predict the components of an ingredient, such as the name of the ingredient, the quantity and the unit. It performed better than the benchmark CRF model when using fewer data examples, and on the full dataset performed similarly but was useful for identifying issues in the data annotations. It also has some success even on languages that it wasn't trained on such as French, Hungarian, and Russian.

Stanford NER is a good implementation of a Named Entity Recognizer (NER) using Conditional Random Fields (CRFs). CRFs are no longer near state of the art for NER, having been overtaken by LSTM-CRFs, which have since been overtaken by Transformer models. However CRFs are still a reasonable baseline, and Stanford NER is used in many papers which is good for reproducibility. It doesn't have a Python binding (a CRF library that does is CRFsuite), but with some work we can train and test a model in Python.

TextRank (Mihalecea and Tarau, 2004) is the idea of using graph ranking algorithms, like PageRank, as an unsupurvised way of extracting key units of text from a document. The interesting part is how they define graphs on the units of text; filtering text units (e.g. words, sentences) to obtain a set of vertices, and using proximity or similarity measures for defining a graph. They specifically look at the applications of keyword extraction and summarisation.

Calculating binomial proportions by group is inherently noisy for small groups. The standard deviation for a group is \(\sqrt{\frac{p(1-p)}{N}}\), where \(p\) is the true proportion of the group and \(N\) the size of the group. The numerator is always at least as large as the distance to 0 or 1, so for small \(N\) the uncertainty is very large relative to the measurement. The resolution of the measurement is at most \(1/N\); with \(N=2\) the only possible outcomes are 0%, 50% or 100%.

Last weekend I read the first part of Kent Beck's Test Driven Development: By Example. He works through a simple example of programming with Test Driven Development in excruciating detail. It shows how you can move in small steps when things get tricky, which is often better than a long debugging session. I typically develop code iteratively in a Jupyter Notebook or IPython REPL. I can pass the code I'm writing and interactively inspect it to make sure it works as inspected.

The Internet Archive's Wayback Machine is a digital archive of a large portion of the internet (hundres of billions of web pages). However they don't store the webpage in its original form, but make some changes to the page to make it easier to view as it was at the time; for example replacing links to images, CSS and Javascript with their archived versions. But how exactly do they change the HTML, and how do we get the original version?

I've been using Linux on laptops for over a decade and have got used to some of the rough edges. When I boot my current laptop connected to an external monitor xrandr will extend the displays between the external monitor and the laptop screen. I have my laptop half-shut low on my desk and so the screen is unusable, and so I open up lxrandr, which always opens up on the laptop screen, and so I awkwardly crane my neck and wiggle the mouse until I click the setting to turn off the laptop screen and press apply, then rush to press the confirmation button.

I've been trying to use pagination with the Internet Archive's CDX for the Wayback Machine but have been getting lots of empty results. The reason is that filters are applied after pagination, and so when using tight filters almost all records will be empty. Using the CDX API Following the documentation it's easy enough to make a query (and similar, but not the same, as Common Crawl's CDX Index). Here's some logic to get captures from en.

Every now and then when previewing Hugo (via hugo serve) as I'm editing it in Emacs I'll get a strange error like: ERROR 2021/10/18 19:36:03 process: readAndProcessContent: walk: Readdir: decorate: lstat /home/user/skeptric/content/user@machine.2139:12345 no such file or directory Often I can work around it by editing and saving the file I've been editing again, but I'll have to restart the Hugo process. However the underlying cause is lockfiles in Emacs, and the easiest fix is to run the elisp (in 24.

I'm currently streamlining the process of building a dataset from web data. I want to make it easy for anyone to build their own dataset in a few hours, which requires making the process as smooth as possible. One thing that makes a major difference is caching intermediate results; it makes the process much faster when you hit an error. But done naively it can be confusing; if you change a function you want the cache to be recalculated.

I'm currently developing a workflow for extracting data from captures of web pages, leveraging large archives like Common Crawl and the Wayback Machine. A pain point in extracting data from external sources is you have to reverse engineer the schema, and quite often it breaks on certain data. To reduce the pain of hitting these errors it makes sense to cache the results to reduce the time of re-extracting the data; especially if it's doing something expensive like parsing a large HTML document.

I recently watched the ACEMS public lecture Optimal Decision Making: A tribute to female ingenuity, where Alison Harcourt (née Doig) talked about her contribution developing Branch and Bound (William Cook has a paper on this history, and the landmark paper was An Automatic Method of Solving Discrete Programming Problems (Land and Doig, 1960)). They talked about how impactful the work was on industry, particularly with logistics where you often have constrained resources (many indivisible) that you need to optimise some goal.

Stein's Paradox states that when trying to estimate the 3 or more means of normally distributed data together, it's always better (on average) to shrink the estimates. Specifically if you've got p independent normally distributed variables \(X_i \sim N(\theta_i, 1) ;\, i=1,\ldots,p\) the best estimates for minimising the mean squared error of all the estimates isn't the values themselves \(X\), and the James-Stein estimator is better (has strictly lower risk).

Estimating a Bernoulli Probability Suppose we want to know the probability of an event occurring; it could be a customer converting, a person contracting a disease or a student passing a test. This can be represented by a Bernoulli Distribution, where each draw is an independent random variable \( \gamma_i \sim {\rm Bernoulli}(\theta) \). The only possible values are failure (represented by 0) with probability \(\theta\) and success (represented by 1) with probability \(1-\theta\) (although the labels are completely arbitrary and we can switch them by setting \(\eta_i = 1 - \gamma_i\) then \(\eta_i \sim {\rm Bernoulli}(1-\theta) \)).

The concept of a multilevel model, also called a mixed effects model or a hierarchical model, is reasonably new to me. It's not the kind of thing typically taught in physics (where there are very explicit models) or in machine learning, but is quite common in social science. I first came across it through Lauren Kennedy on the Learning Bayesian Statistics Podcast, through talking with a trained neuroscientist and a trained statistician who were talking about fixed and variable effects as I went cross-eyed, and through the excellent Regression and Other Stories textbook which makes many allusions to it (to be expounded on in their upcoming sequel Applied Regression and Multilevel Models).

Suppose you want to translate text from one language to another. Most people's first point of call is an online translation service from one of the big cloud providers, and most translation libraries in Python wrap Google translate. However the free services have rate limits, the paid services can quickly get expensive, and sometimes you have private data you don't want to upload online. An alternative is to run a machine translation model locally, and thanks to Hugging Face it's pretty simple to do.

High cardinality categorical data are tricky for machine learning models to deal with. A linear model tries to estimate a different coefficient for every category, treating them as totally independent. There are tools like hierarchical models that can encode some cross-correlations but (to my knowledge) they don't scale well to large datasets. A tree model will try to estimate different coefficients on groups of categories (based on the order they are sorted in), but for many categories there is no canonical order (like locations on a map).

This is the third on a series of articles showing the basics of building models in Stan and accessing them in R. Now that we can specify a linear model and fit it in with formula syntax, and specify priors for the model, it would be useful to be able to make predictions with it. In principle making predictions from our linear model \( y \sim N(\alpha + \beta x, \sigma)\) is easy; to make point predictions we take central estimates of the coefficients \(\hat{\alpha}\) and \(\hat{\beta}\) and estimate \( y \approx \hat{\alpha} + \hat{\beta} x\).

This is the second on a series of articles showing the basics of building models in Stan and accessing them in R. In the previous article I showed how to specify a simple linear model with flat priors in Stan, and fit it in R with a formula syntax. In this article we extend this to specify priors; defaulting to general weakly informative priors but allowing use of specific priors.

I wanted to fit a Bayesian Tobit model, but I couldn't find one (probably because I didn't know how to look). So I decided to build one in Stan, which I had never used before. This article is the first in a series showing how I got there; this one builds a linear model in Stan and makes it useable from R using formula syntax, then next we add priors to the model, make model predictions from R, and then handle censored values with Tobit regression.

I was working through problem 15.5 of Regression and Other Stories, which asks to fit a probit regression to a previous example with a logistic regression. I used a model I had built on the National Election Survey dataset (on rstanarm 2.21.1): fit_nes_probit <- rstanarm::stan_glm(rvote ~ income_int_std + gender + race + region + religion + education_cts + advanced_degree + party + ideology3 + gender : party, family=binomial(link="probit"), data=nes92) When I got this error about the chains not converging:

This is the essence of the refactoring process: small changes and testing after each change. If I try to do too much, making a mistake will force me into a tricky debugging episode that can take a long time. Small changes, enabling a tight feedback loop, are the key to avoiding that mess. Martin Fowler, Refactoring, Improving the Design of Existing Code You've got a Python analytics process and have to make a change to how it works.

In Bayesian statistics you have to choose a prior distribution for the parameters to combine with the data to get a posterior distribution. Choosing a tight prior, assuming that the parameters should live in a particular space, reduces the impact of the data on the posterior estimates. This is just like regularisation in machine learning where adding a penalty to the loss function prevents over-fitting. This is more than just an analogy, and this article will explore a couple of cases with constant regression and classification.

When understanding how a binary prediction depends on a continuous input I find a very useful way is to bin it into quantiles and plot the average probability. For example here's a plot using the iris dataset showing how the probability that a flower is a "virginica" and not a "versicolor" changes with the sepal width of the flower. This kind of plot can show nonlinearities and indicate how we should include this variable in a logistic regression.

The best ways to build a skill is to practice it, and data analytics is no different. That means you need to find datasets to analyse, and luckily there are plenty of open datasets available. There's always more ways to find data sets, but here's a few places to get inspiration. I'm looking for a particular dataset Google has a dataset search which is a great place to start. I'm looking for inspiration Data in brief is an Open Journal for publishing datasets, with descriptions Kaggle hosts many datasets as well as their competition datasets which are great to benchmark Machine Learning techniques, and many have notebooks showing exploration of the datasets Tidytuesday has weekly datasets for exploring, and you can look at other's explorations Jeremy Singer-Vine's Data is Plural has weekly interesting datasets.

A couple of months ago I transitioned into the role of data science lead. Suddenly instead of just having to think about how to deliver new solutions, improve operational processes and review contributions I had to also mentor a new team member, develop new team processes and rituals, coordinate and project manage, shape OKRs, and contribute to developing presentations to executives to approve what we were delivering. For a few weeks it was like I was working two jobs; in the day I would either be in meetings or working on planning and communication, and at night I would be modelling or writing code to make sure we could deliver the right solution on time.

I’m currently reading Regression and Other Stories which contains lovely plots of coefficients and their distributions. What really impressed me is how easily I could solve this with the few concepts in the tidyverse. Suppose we’ve got an rstanarm model like this: model <- rstanarm::stan_glm(Petal.Width ~ Sepal.Length + Sepal.Width + Species, data=iris, refresh=0) We can access all the coefficients from the posterior draws using as.matrix. With a few standard transformations we can plot the distribution of each of the coefficients.

Interpretable models are incredibly useful for getting users to trust your model (and understand when not to trust it). In projects helping subject matter experts make decisions it helps them see what's going into it, and what adjustments need to be made. However a large part of what makes a model interpretable is not just its form, but how it's presented. Reparameterising it in terms that make sense from a decision makers point of view can make all the difference in gaining insight, trust and adoption.

In Noise by Kahneman, Sibony and Sunstein, they discuss how much variation there is in sentences for very similar circumstances. This made me curious how much variation in sentencing there was in the Australian legal systems. In Victoria the sentencing council publishes statistics on sentences by offence that helps answer this question. But I'd recently seen some news about South Australia, and was wondering if I could get the information for South Australia.

I have been trying to extract job ads from Common Crawl, and have designed a pipeline with 3 phases; fetching the data, extracting the content and normalising the data. However the way I implemented this is using inheritance to remove some of the duplication between the steps. While this made it quick to start it's hard to understand, and to extend it I need to introduce more duplication. My current design started pragmatically; a data source needed to be able to fetch, extract and normalise.

Many analytics codebases consist of a pipeline of steps, doing things like getting data, extracting features, training models and evaluating results and diagnostics. The best way to structure the code isn't obvious and if you're having trouble importing files, getting module not found errors or are tinkering with PYTHONPATH it's likely you haven't got it right. A way I've seen many data analytics processing pipelines structured is with a series of numbered steps:

Typer is a nice application for succinctly building Python CLIs built on top of Click. However when you've got subcommands they're listed in alphabetical order. It would be nice to have the commands ordered in the same way they are in the code, so you can present them to the user in the clearest order to read. There's a simple solution for this in Python 3.6+ (for older versions outside of C Python you may need a more elaborate version with OrderedDict).

Property based testing is a really useful technique where you state a property about your code and then verify it with random data. The difficulty is generating good random data that will thoroughly exercise your code. For text data you can do this by generating text, in particular with regular expressions. While you could easily do this with hypothesis from_regex strategy you could roll your own with Hypothesis and the library rstr as follows:

Gradient descent is an effective algorithm for finding the local extrema of functions, and the global extrema of convex functions. It's very useful in machine learning for fitting a model from a family of models by finding the parameters that minimise a loss function. However sometimes you have extra constraints; I recently worked on a problem where the maximum value of the fitted function had to occur at a certain point.

I used to think the whole point of software verifications like types and tests was to ensure a piece of software worked as specified. Consequently if a piece of software already worked there wasn't much point in adding automated tests; sure we might find a few edge cases that didn't work, but we already would have had the ones that impacted end users in bug reports. I now think the primary benefit of verifications is about making software easier to change without losing quality by introducing regressions.

When I tried to install tidyverse in WSL2 I ran into issues with timedatectl and xml2. The simple solution is: # Assuming Debian derivatives sudo apt-get install libxml2-dev # Modify TZ to whatever your timeozne is TZ="Australia/Sydney" R -e 'install.packages("tidyverse")' What happens When I try to install tidyverse I get this error: > install.packages('tidyverse') ERROR: configuration failed for package ‘xml2’ System has not been booted with systemd as init system (PID 1).

I am a very recent convert on automatic refactoring tools. I thought it was something for languages like Java that have a lot of boilerplate, and overkill for something like Python. I still liked the concept of refactoring, but I just moved the code around with Vim keymotions or sed. But then I came up against a giant Data Science codebase that was a wall of instructions like this: import pandas as pd import datetime df = pd.

I've been writing some ARM Assembly as part of a Raspberry Pi Operating System Tutorial, and writing in Assembly really forces me to think about performance in terms of registers and instructions. When I'm writing Python trying to write concise code leads to breaking a problem into small functions or methods (and using idioms like list comprehensions). The same drive for concise code in Assembly leads me to reduce the number of instructions used and the number of registers, but even though it feels like its making things more efficient it may have negligible actual impact.

I've been revisiting baremetal Raspberry Pi programming (with Alex Chadwick's Baking Pi tutorial, although there are plenty of others). It really highlights how much I take for granted. I spend a lot of time in languages like Python and R processing data, without much understanding of the interpreters written in C, let alone thinking about the compilers that turn those interpreters and libraries into assembly, or how that assembly executes.

Tabular data in Pandas is very flexible, but sometimes you just want a key value store for fast lookups. Because Python is slow, but Pandas and Numpy often have fast C implementations under the hood, the way you do something can have a large impact on its speed. The fastest way I've found to convert a dataframe to a dictionary from the columns keys to the column value is: df.set_index(keys)[value].to_dict() The rest of this article will discuss how I used this to speed up a function by a factor of 20.

Modern Javascript web frameworks often embed the data used to render each webpage in the HTML. This means an easy way of extracting data is capturing the string representation of the object with a pushdown automoton and then parsing it. Sometimes Python's json.loads won't cut it for dynamic JSON; one option is demjson but another much faster option is chompjs. Chompjs converts a javascript string into something that json.loads. It's a little less strict than demjson; for example {"key": undefined} will be converted by chompjs.

One of my favourite tools in Pandas is agg for aggregation (it's a worse version of dplyrs summarise). Unfortunately it can be difficult to work with for custom aggregates, like nth largest value. If your aggregate is parameterised, like quantile, you potentially have to define a function for every parameter you use. A neat trick is to use a class to capture the parameters, making it much easier to try out variations.

I use Vim keybindings in Emacs through Evil Mode and Evil Collection. Often I'll copy something, make some edits, and then want to paste the text. The problem is that the edits changed what was on the register and I want to recover that text. In Vim (and Evil mode) I can type :reg to see what's in the registers, find the value I want to paste, commit the register to memory (for example 8), exit and then paste that register ("8p from normal mode, C-r 8 from insert mode).

Dictionaries in Python (in other languages called maps or hashmaps) are a useful and flexible data structure that can be used to solve lots of problems. Part of their charm is the affordances in the lanugage for them; setting and accessing with square brackets [], deleting with del. But sometimes you want a dictionary that persists across sessions, or can handle more data than you can fit into memory - and there's a solution persistent dictionaries.

Scrapy is a fast high-level web crawling and web scraping framework. But as much as I want to like it I find it very constraining and there's a lot of added complexity and magic. If you don't fit the typical use case it feels like a lot more work and learning doing things with scrapy than without. I really like Zyte (formerly ScrapingHub) the team behind Scrapy. They really know what they're talking about with great blogs about QA of Data Crawls, guide to browser tools, how bots are tracked, and Scrapy's documentation has a very useful page on selecting dynamically-loaded content.

The internet is full of useful information just waiting to be collected, processes and analysed. While getting started scraping data from the web is straightforward, it's easy to tangle the whole process together in a way that makes it fragile to failure, or hard to change with requirements. And the internet is inconsistent and changing; in anything but the smallest scraping projects you're going to run into failures. I find it useful to conceptually break web scraping down into four steps; selecting the data to retrieve, fetching the data from the internet, extracting structured data from the raw responses, and watching the process to make sure it's functioning correctly.

I find taking screenshots in Linux a bit painful. My current way is to use GIMP to create an image from a screenshot, but it's a bit slow to startup and interrupts my flow. I've had trouble installing Shutter which I haven't worked through yet. However I've just found out that Firefox has a way to take screenshots. All you need to do is press Control-Shift-S and then it brings up a selector where you can pick an element, or a region (like an improved version of Windows Snipping tool).

You can use Python to read, process and manage your emails. While most email providers provide autoreplies and filter rules, you can do so much more with Python. You could download all your PDF bills from your electricity provider, you could parse structured data from emails (using e.g. BeautifulSoup), sort or filter by sentiment, or even do your own personal analytics like Steven Wolfram. The easiest tool I've found for reading emails in Python is imap_tools.

I sometimes build hobby machine learning APIs that I want to show off, like whatcar.xyz. Ideally I want these to be cheap and low maintenance; I want them to be available most of the time but I don't want to spend much time or money maintaining them and I can have many of them running at the same time. My current solution is Cloud Compute (e.g. Digital Ocean or Linode) which has low fixed costs (around $5 per month).

A common question with conversions and other rates, is how big a sample do you need to measure the conversion accurately? To get an estimate with standard error \(\sigma\) you need at most \( \frac{1}{4 \sigma^2} \) samples. In general if the true conversion rate is p it is \(\frac{p(1-p)}{\sigma^2}\). So let's say we want to measure the conversion rate within about 5%. To be conservative we'd want the standard error to be a bit less than that, say 3% Then we would need at least \( \frac{1}{4 (0.

Suppose you've got two dice; what's the probability the sum of their rolls will add up to 4? You simply look at all the ways the values of the dice could sum to 4 (e.g. 1 and 3, 2 and 2 or 1 and 4), and add up their probabilities (in this case each is 1/36, so totalling 3/36 or 1/12). You could use a probability square to visualise this calculation.

What sample size do you need to capture an effect with 95% confidence 80% of the time? For a normal/binomial distribution the answer is roughly \( \left(2.8 \frac{\sigma}{\epsilon}\right)^2 \), where \( \sigma \) is the standard deviation of the data and \( \epsilon \) is the size of the effect. The ratio \( \frac{\sigma}{\epsilon} \) says the smaller the effect size is relative to the variability in the data the larger the sample size you will need.

When running an A/B test the sample sizes can seem insane. For example to observe a 2 percentage point uplift on a 60% conversion rate requires over 9,000 people in each group to get the standard 95% confidence level with 80% power. If you've only got less than 18,000 customers you can reach, which is very common in businesss to business settings, it's impossible to conduct this test. But if you look in terms of the outcomes, doing an A/B test on a few hundred users may actually greatly increase your outcomes.

I have a personal CV written in TeX. I wouldn't use TeX again today, but I've kept it maintained and haven't had reason to migrate it. Unfortunately the dependencies can be painful; whenever I move to a new machine I need to remember which packages I need to install. A Dockerfile is a handy way to wrap up the dependencies, and create a container with those dependencies. Docker is a really handy tool for dependency management.

Sometimes it's useful to rerun a task whenever a file changes; whether that's a linter or a test suite, or a preview. I recently wanted to recompile a TeX file to PDF whenever I saved a change, and it was easy with inotify, using instructions from superuser. To install inotify on a Debian derivative you can use sudo apt install inotify-tools. Then you can set it to run a command whenever a file is done saving.

Suppose you have a long standing heart condition, and are considering undergoing a surgical procedure that could alleviate the procedure, but has its own set of risks. You happen to have a good friend who's a statistician at the hospital you're being seen at and can get you historical frequencies of complications. However she asks you what reference set do you want to use? Do you want complications related to just the specific procedure treating your condition, or for all procedures on that area of the heart for a variety of conditions?

A large number of the physicists I trained with are now data scientists, and it's not uncommon to meet a data scientist who trained in Physics. Part of this is because there's not a lot of physics jobs, especially in Australia. But another reason is that the training we get as physicists is very similar to what you need for data science. David Bailey, a physicist from the University of Toronto, has objectives for their undergraduate Physics program which describes "the Way of the Physicist":

When working with distributions that were powers of exponentials, of which the normal and exponential distributions are special cases, I had to calculate the integrals of exponentials. It's possible to transform these into expressions involving the Gamma function. Specifically I found that for all positive p: \[ \int_{0}^{\infty} x^m e^{-x^p}\, \rm{d}x = \frac{\Gamma\left(\frac{m+1}{p}\right)}{p} \] This is useful for calculating moments of powers of exponentials, namely for positive p and k:

When evaluating binary classifier (e.g. will this user convert?) the most obvious metric is accuracy; what's the probability a random prediction is correct. One issue with this metric is if 90% of the cases are one class a high accuracy isn't really impressive; you need to contrast it with a constant model predicting the most frequent class. More subtly it's not a very sensitive measure, by measuring cross-entropy of predicted probabilities you get a much better idea of how well your model is working.

As a professional your reputation is very important to your career success. To get people to offer you work, to pay for your advice or to buy products from you they need to trust that you will deliver them value. The most common heuristic for this is your reputation; what other people say about you, what you have done and what certifications you have. It's important that your reputation is very specific to the kind of work you want others to buy.

When creating a batch process you typically add logging statements so that when something goes wrong you can more quickly debug the issue. Then when something goes wrong you either try to fix and rerun it, or otherwise run the process in a debugger to get more information. For many tasks Jupyter Notebooks are better for these kinds of batch processes. Jupyter Notebooks allow you to write your code sequentially as you usually would in a batch script; importing libraries, running functions and having assertions.

Whenever I use Jupyter Notebooks for analysis I tend to set a bunch of options at the top of every file to make them more pleasant to use. Here they are for Python and R with IRKernel Python # Automatically reload code from dependencies when running cells # This is indispensible when importing code you are actively modifying. %load_ext autoreload %autoreload 2 # I almost always use pandas and numpy import pandas as pd import numpy as np # Set the maximum rows to display in a dataframe pd.

Sometimes I have nested object of dictionaries and lists, frequently from a JSON object, that I need to deal with in Python. Often I want to load this into a Pandas dataframe, but accessing and mutating dictionary column is a pain, with a whole bunch of expressions like .apply(lambda x: x[0]['a']['b']). A simple way to handle this is to flatten the objects before I put them into the dataframe, and then I can access them directly.

Indexing into nested objects of dictionaries and lists in Python is painful. I commonly come up against this when reading JSON objects, and often fields can be omitted. I haven't found a solution to this and so I've invented a tiny DSL to do this. It works like this: d = [{'a': [{'b': 'c'}, {'d': ['e']}]}] assert extract(d, '0.a.1.d.0') == d[0]['a'][1]['d'][0] assert extract(d, '1.a.1.d.0') == None You can specify a path into an object, separated by periods, and it will extract it returning None if that path doesn't exist.

Nbdev is a tool to make it possible to develop Python libraries in Jupyter notebooks. At first I found this idea scary, but after watching the talk I like Notebooks and seeing how it works I think it's got the best of all worlds. It lets you put code, documentation, examples and tests all together in context and provides tooling to extract the code into an installable library, run the tests and produce great hyperlinked documentation.

Probabalistic language models can be used directly as a classifier. I'm not sure if this is a good idea; in particular it seems less efficient than building a classifier, but it's an interesting idea. A language model can give the probability of a given text under the model. Suppose we have multiple language models each trained on a distinct corpus representing a class (e.g. genre or author, or even sentiment). Then we can calculate the probability conditional on that model and compare them to calculate the class.

An N-gram language model guesses the next possible word by looking at how frequently is has previously occurred after the previous N-1 words. I think this is how my mobile phone suggests completions of text; if I type "I am" it suggests "glad", "not" or "very" which are likely occurrences. To make everything add up you have to have special markers for the start and end of the sentence, and the I think the best way is to make them the same marker.

The Gulag Archipelago is a singular piece of literature about the horrors of arbitrary arrest, inhumane interrogations, prolonged imprisonments, deadly work camps and exile that impacted tens of millions of people in the first half of the twentieth century. Aleksandr Solzhenitsyn has a way of conveying these horrors in a truly compelling way; somehow the descriptions of torture are close and detailed enough to be vivid, varied enough to be somewhat comprehensive, yet not repetitive nor gratuitous.

Today I learned that Dr Rodney James Crewther, known affectionately by colleagues and students as Rod, died last week. I only knew Rod through my education at the University of Adelaide in the late noughties, where he was a senior lecturer in the Physics department specialising in particle physics, but he had a huge impact on me. My thoughts are with those close to him in this difficult time. Rod truly cared about educating students.

Once a problem becomes moral, the acceptable solution space collapses. I'm reading Sylvia Nasar's book Grand Pursuit, and she talks about how Malthus' An Essay on the Principal of Population made the argument that as the poor classes gained more money they would always reproduce in greater numbers until they reached their previous state of poverty. This and other contemporary works of political economy posed these kinds of arguments as natural laws.

Salary ranges come in many forms; how can we convert them to a common form? A first approximation is to annualise them; it ignores the difference between full-time, part-time, and temporary work. The other question is how to pick the range, for jobs with a bery large range. I started with the minimum because the maximum is often an inspirational nubmer (especially in commission sales roles). The way I approached this was:

Johnathan Haidt's The Righeous Mind: Why Good People are Divided by Politics and Religion is about the moral norms of groups. As someone not familiar with moral psychology I found the book discussed many interesting ideas I wasn't aware of, but didn't provide much evidence for Haidt's own theories and claims. The book is reasonably well written with good structure and some excellent metaphors, but sometimes goes on unnecessarily long detours into the author's personal life and the repetition can be wearing.

When a program primarily sources and transforms data then copyleft licenses add very little protection over other open source licenses. Because of this I've licensed my open data processing code as MIT because more complex licenses would prevent other people from using it, without adding much sharing. There are three main license types that are used in Open Source; MIT, Apache and GPL (with BSD family somewhere between MIT and Apache).

When running the Kaggle API method dataset_list_files in a Jupyter notebook I got an error about __repr__ returning a non-string. At first I thought the function was broken, but then I realised it was just how it was displaying in Jupyter that was breaking because the issues were all in IPython: --------------------------------------------------------------------------- TypeError Traceback (most recent call last) IPython/core/formatters.py in __call__(self, obj) 700 type_pprinters=self.type_printers, 701 deferred_pprinters=self.deferred_printers) --> 702 printer.pretty(obj) 703 printer.

When all you've got is the hammer of mathematics, everything looks like an optimisation problem, you just need to choose the right objective function. So what should the objective function be for life? People today in general have much more means and much more freedom (i.e. fewer constraints in the solution space) than many of their ancestors, so what should we optimise? A study from Daniel Kahneman and the economist Angus Deaton says High income improves evaluation of life but not emotional well being.

When you solve a problem in code you will use some programming approach, and the approach you choose can make a big impact on your efficiency. I talk about approach rather than language because it's more than just the language. A project will typically only use a subset of the language (especially for massive languages like C++), some set of libraries, and develop patterns in the lanugage for working with those libraries.

Something that has always bothered me about pip in Python is that you would get errors about inconsistent packages. Things still seemed to work surprisingly often, but it meant that the order you installed packages could lead to very different results (and one ordering may cause your test to fail, even if that doesn't succeed). Now there is a new resolver in Pip 20.3 for pip that checks the dependencies and tries to find versions that meet all constraints.

The Mere Exposure Effect says that people will prefer something they've seen or heard multiple times before than something less familiar. The Robert Zajonc paper Attitudinal Effects of Mere Exposure covers this topic comprehensively. It includes experiments showing people like foreign words, chinese characters, photographs and nonsense words they have been exposed to more than if they had been exposed less. There are other aspects that impact affect more than mere exposure; like people prefer pronounceable words or photographs of smiling people, but exposure almost always helps.

I was recently trying to run a pipenv script, but it gave an error that it required Python 3.7 which wasn't installed. Unfortunately I was on Ubuntu 20.04 which has Python 3.8 as default, and no access to earlier versions in the repositories. However pipenv gave a useful hint; pyenv and asdf not found. The asdf tool allows you to configure multiple versions of applications in common interactive shells (Bash, Zsh, and Fish).

In mathematics it's surprising how often something that's obvious (or trivial) to someone else can be revolutionary (or weeks of work) to someone else. I was looking at the annoy (Approximate Nearest Neighbours, Oh Yeah) library and saw this comment: Cosine distance is equivalent to Euclidean distance of normalized vectors I hadn't realised it at all, but once the claim was made I could immediately verify it. Given two vectors u and v their distance is given by the length of the vector between them: \( d = \| u - v \| = \sqrt{(u - v) \cdot (u - v)} \).

When I started as a professional developer there were a number of competing version control systems. However Git seems to have almost entirely won this battle. One of the most popular centralised version control systems is Subversion (SVN), which was largely an improvement of Concurrent Versioning System (CVS). But Distributed Version Control Systems, starting with Git became really popular. With a centralised system you have to lock files on the central server when editing and unlock them when you're finished, to make sure no one else interferes with your work.

Sometimes you want to feed a bunch of files to a program, and this is often easily done with find and xargs. Suppose you have an executable doit that you want to execute on all Python files in src/; you can do this directly with find: find src/ -name '*.py' -exec doit {} \; You can use xargs for this as well; but if there's a chance that a path could contain a space somewhere it's best to use -print0 with find and -0 with xargs to separate all arguments with nulls (rather than spaces):

I’ve written a derivation of how to find the centroid of a polygon on a sphere. This post shows it explicitly in numerical computations, and also looks at the solution in Spherical Averages and Applications to Spherical Splines and Interpolation, by Buss and Fillmore, ACM Transactions on Graphics 20, 95–126 (2001). Explicitly coding mathematics is a great exercise; having to concretely represent everything unearthed gaps in my understanding and found errors in both drafts of my derivation and the paper.

You join a new team and your first task is to run the monthly batch process. It transforms some data, trains a business critical model, and outputs some reporting. Your coworker who is leaving the team talks you through the process and what you need to do. The problem is that it's a bunch of scripts and ad hoc SQL that breaks all the time and has to be manually patched over.

Can we have an interpretable model that has as good performance as blackbox models like gradient boosted trees and neural networks? In a 2020 Empirical Methods for Natural Language Processing Keynote, Rich Caruana says yes. He calls interpretable models glassbox machine learning, in contrast to blackbox machine learning. It is models in which a person can explicitly see how they work, and follow the steps from inputs to outputs. This interpretability is subtly different from explainable (explainable to who?

I want to find examples of other Hugo blogs, but they're not really easy to search for. Unless someone put "Hugo" in the descrption (which is actually common) there's no real defining files. However there's a set of files that are in a lot of Hugo Blogs and we can search them in Github with the GHArcive BigQuery Export. The strategy is that most Hugo blogs will contain a /themes folder, a /content folder a /static folder and a config.

R core looks like it's getting a new pipe operator |> for composing functions. It's just like the existing magrittr pipe %>%, but has been implemented as a syntax transformation so that it is more computationally efficient and produces better stack traces. The pipe means instead of writing f(g(h(x))) you can write x |> h |> g |> f, which can be really handy when changing dataframes. Python's Pandas library doesn't have this kind of convenience and it opens up a class of error that won't happen in that R code.

Cosine similarity is often used as a similarity measure in machine learning. Suppose you have a group of points (like a cluster); you want to represent the group by a single point - the centroid. Then you can talk about how well formed the group is by the average distance of points from the centroid, or compare it to other centroids. Surprisingly it's not much more complex than finding the geometric centre in euclidean space, if you pick the right coordinate system.

I recently convinced myself that I was right and a published paper was wrong. I did all the calculus by hand and decided they must have missed something. I resolved it by calculation. I played some very basic examples in my head and thought I was right, but I couldn't be sure. So I quickly implemented a concrete example case in Python using numpy. This helped me resolve pretty quickly that I was wrong; some searching produced a counterexample.

When people access products online their behaviour gives lots of information about both the people and the products. This information deeply enriches understanding of how to better serve your customers, how your products are related to each other and can help answer deeper questions about them. However you need to find a way to unlock the information. Using behavioural information can greatly improve modelling on the tabular data in your database.

The Critical Assessment of protein Structure Prediction (CASP) runs every two years to predict the shape of a protein, the building blocks of life, from its sequence of amino acids. We know the shape of a bunch (around 170,000) of proteins from techniques like X-ray crystallography and Magnetic Resonance Imaging, but it's a big experimental job to actually measure this. However we know the sequence of millions of proteins due to cheap DNA sequencing and the DNA to protein translation.

I often use method chaining in pandas, although certain problems like calculating the second most common value are hard. A really good solution to adding custom functionality in a chain is Pandas pipe function. For example to raise a function to the 3rd power with numpy you could use np.power(df['x'], 3) But another way with pipe is: df['x'].pipe(np.power, 3) Note that you can pass any positional or keyword arguments and they'll get passed along.

I was reading [Paul Graham's How to Think For Yourself] where he talks about independent-mindedness. His examples are scientists, investors, startup founders and essayists as professions where you can't do well without thinking differently from your peers. While there's some useful commentary in the article about how to broaden your worldview, I don't think a really independent minded person would do well in those professions. To succeed in science you need to study something people are interested in.

Analytics projects are messy. It's rarely clear at the start how to frame the business problem, whether a given approach will actually work, and if you can get it adopted by your partners. However once you have a framing the modelling part can be iterated on quickly by structuring the project like a Kaggle Competition. The modelling part of analytics projects will go smoothly only if you have clear evaluation criteria.

Type hints in Python allow statically verifying the code is correct, with tools like mypy, efficiently eliminating a whole class of bugs. However sometimes you get the message found module but no type hints or library stubs, because that library doesn't have any type information. It's easy to work around this by adding type stubs. When you see this error it's worth first checking that there aren't any types already available.

I've been trying to extract job ads from Common Crawl. However I've been stuck on how to structure the code. Thinking through the relationships really helped me do this. The architecture of the pipeline is a set of methods that fetch source data, extract the structured data and normalise it into a common form to be combined. I previously had these methods all written in one large file, adding each extractor to a dictionary, which was a headache to look at.

I often have way too many Jupyter notebook tabs open and I have to distinguish them from the first couple letters of the notebook in front of the Jupyter organge book icon. What if we could change the icons to visually distinguish different notebooks? I thought I found a really easy way to set the icon in Jupyter notebooks... but it works in Firefox and not Chrome. I'll go through the easy solution works in more browsers and the hard solution.

When looking at Pandas dataframes in a Jupyter notebook it can be hard to find what you're looking for in a big mess of numbers. Something that can help is formatting the numbers, making them shorter and using graphics to highlight points of interest. Using Pandas style you can make the story of your dataframe standout in a Jupyter notebook, and even export the styling to Excel. The Pandas style documentation gives pretty clear examples of how to use it.

I've finally built a first iteration of a job extraction pipeline in my job-advert-analysis repository. There's nothing in there that I haven't written about, but it's simply doing the work to bring it all together. I'm really happy to have a full pipeline that extracts lots of interesting features to analyse, and is easy to extend. I've already talked about how to extract jobs from Common Crawl and the architecture for extracting the data.

I tried to run a shell script and got this error: set: Illegal option -o pipefail I had a quick look and the first line was #!/bin/sh, the -o pipefail isn't valid across POSIX shells so I would expect that to fail. More specifically on modern Ubuntu /bin/sh is dash which doesn't support these bash like constructions. But /bin/sh is very different on different systems; on some it is bash, on others it's ash (from which dash is derived), and on others it's ksh or something else.

Remember after you run the update process to run the fix script on the production database. But run it twice because it only fixes some of the rows the first time. Oh, and don't use the old importer tool in the import directory, use the one in the scripts directory now. You already used the old one? It's ok, just manually alter the production database with this gnarly query. Ah right, I see the filler table it uses is corrupted, let's just copy it from a backup.

I was listening to the $100 MBA Podcast about How to Easily Create a Small Information Product. I really like the idea of building a tiny product in under 12 hours of work to get experience making something and building success in small steps. Their basic premise is it's really easy to create a simple product that let's you share your expertise to help people make a small transformation: Create a one-page infographic or cheatsheet in Powerpoint and export it as a PDF Write a small ebook in a Word processor and convert it to PDF (or an ebook with Pandoc) Create some small videos using any HD camera and a USB microphone Then you can easily use Gumroad for fulfilment, even for a free product.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.11 Estimate the energy in a 9-volt battery. Is it enough to launch the battery into orbit? I have already (mis)estimated the energy of a battery, but looked it up as 500 mAh. Energy density required to launch into space To launch into space you have to exchange energy to counteract the change in gravitational energy (at least, you'll need more for air resistance).

This is from Sanjoy Mahajan's The Art of Insight Problem 1.11 Estimate the energy in a 9-volt battery. Is it enough to launch the battery into orbit? We're just going to estimate the first part. Battery Energy A volt is energy per unit charge \( V = \frac{E}{q} \). To get towards an energy we need an amount of charge; the current in Ampere is the charge per unit time \( I = frac{q}{t} \).

A lot of times I've failed by biting off more than I can chew. I get in over my head and lose motivation. A lot of times I've succeeded it's by starting small and slowly building up a roll of successes. When I was in highschool I tried to build a simulation of the solar system for a project. I wasn't satisfied with building ellipses, I wanted to take into account all the N-body interactions.

My Windows laptop was halting to a crawl; I was waiting seconds to switch windows and even typing took a couple of seconds to respond. I opened the task manager by hitting Ctrl-Shift-Esc and saw that Vmemm was using >95% of my memory. What the heck is Vmemm and how can I stop it using all my memory? Vmemm is the process associated with virtual machines on Windows. I'm using WSL2 and Docker (through WSL2), and so all their memory appears on Vmemm.

You've written your web application or API and you now want to deploy it to a server. You don't want to run it as root, because if someone finds a vulnerability in the server then it will be trivial for them to take over the system. However only root has permission to run applications on ports 80 and 443. There are a few ways to do this, but only a couple that make sense for an interpreted language (like Python, as opposed to a compiled binary).

AWS Athena is a managed version of Presto, a distributed database. It's very convenient to be able to run SQL queries on large datasets, such as Common Crawl's Index, without having to deal with managing the infrastructure of big data. However the downside of a managed service is when you hit its limits there's no way of increasing resources. Today I was running some queries for a regular reporting pipeline in Athena when I got failure with the error Query exhausted resources at this scale factor.

I've been trying to extract job ads from Common Crawl. However I was stuck for some time on how to actually write transforms for all the different data sources. I've finally come up with an architecture that works; download, extract and normalise. I need a way to extract the job ads from heterogeneous sources that allows me to extract different kinds of data, such as the title, location and salary. I got stuck in code for a long time trying to do all this together and getting a bit confused about how to make changes.

I started regularly writing this website to get better at writing, to build a portfolio and share my learnings. Because of this I haven't been focused on building an audience or looking at analytics. However now I've been writing continuously for 6 months I'd see if I learned anything interesting from looking at Google Analytics. I installed Google Analytics a couple of weeks ago on the website to see how people are actually viewing my site.

A couple years ago I built whatcar.xyz which predicts the make and model of Australian cars. It was built mainly with externally sourced data and so only works sometimes, under good conditions. To make it better I've started collecting my own training data. External data sources are extremely convenient for training a model as they can often be obtained much more cheaply than curating your own data. But the data will almost always be different to what you are actually performing inference on, and so you're relying on a certain amount of generalisation.

When programming it's easy to think about the happy path. The path along which you get well-formed valid data, all your requests return successfully and everything works on your target platform. When you're in this mindset it's easy to just check it works in one case and assume everything is alright. But the majority of real work in programming is the unhappy paths. While you always need to be thinking about how things could go wrong, it's much more important in web programming.

Building a web application is great because, if it is well built, it can be accessed across many operating systems. But sometimes you want to access particular aspects of the device; for example take a picture from a mobile camera. It turns out this is easy to do on many systems in HTML. A year ago I built whatcar.xyz which classifies a photo of an Australian car with its make and model.

There's a common misconception that the best way to build up an NLP dataset is to first define a rigorous annotation schema and then crowdsource the annotations. The problem is that it's actually really hard to guess the right annotation schema up front, and this is often the hardest part on the modelling side (as opposed to the business side). This is explained wonderfully by spaCy's Matthew Honnibal at PyData 2018.

Be settled in your life and as ordinary as the bourgeois, in order to be fierce and original in your works. Gustave Flaubert, To Gertrude Tennant (December 25, 1876) It's hard to find the energy and focus to be creative when your life is a mess. Before you can be productive you need to sleep well, eat well, exercise well and have good routines and social supports. See here for more on the origin of this quote.

Experiments reveal the relationship between inputs and outcomes. With statistical methods you can often, with enough observations, tell whether there's a strong relationship or if it's just noise. However it's much harder to know how generally the relationship holds, but it's essential for making decisions. Suppose you're testing two alternate designs for a website. One has a red and green button with a santa hat and bauble, and the other has a blue button.

Static website generators fill a useful niche between handcoding all your HTML and running a server. However there's a plethora of site generators and it's hard to choose between them. However I've got a simple recommendation: if you're writing a blog use Jekyll (if you don't want to use something like Wordpress). Static website generators compile input assets into a set of static HTML, CSS and Javascript files that can be deployed almost anywhere.

There's a common misconception that the best way to get promoted is to excel in your current role. However doing too well in your current role can get you stuck. The actual best way is to grow and start practising the skills for the next role. A friend of mine worked as a night filler at a supermarket, stocking shelves. He worked hard to be one of the best fillers and work his way up.

Sleep is really important. When people don't sleep the get less effective at everything. There are also chronic health patters associated with bad sleep. Worst of all when you're overtired you may not even feel tried. Make sure you get seasonable sleep

Whenever someone gets an idea in their head they start filtering out evidence that contradicts that idea. This idea is called confirmation bias, people start looking for evidence that confirms their current idea and neglecting evidence that challenges it. There's no way to completely beat a bias, but something that helps me is re-framing the question. The first question that comes is normally "Can". Can it be? This leads to looking for evidence that confirms the idea.

One of the hardest parts of learning something new is motivation. This is why one of the best ways to learn programming is editing code; it's goal driven so motivation is built in. I've successfully used this to start learning how to write Hugo themes. Now that I've got a reasonable collection of posts, over 250, I would like to understand what content people are actually accessing on this website to get an idea of what would be useful.

I have been publishing this website using Github Actions with Hugo on push and on a daily schedule. I recently received an error notification via email from Github, and wanted to check whether it was an intermittent error. Unfortunately I couldn't find anyway to rerun it manually; I would have to push again or wait. Fortunately there's a way to enable manual reruns with workflow_dispatch. There's a Github blog post on enabling manual triggers with workflow_dispatch.

How dense is the population in Australia? I've looked at the Gridded Population of the World and you can see that the population is concentrated around the few capital cities on the coast. It's hard to visually average something so lumpy, but it's easy to estimate it. I know it's about 10 hours driving from Melbourne to Sydney, and about the same again to Brisbane. Brisbane is about halfway between Melbourne in the south and Cairns in the far north.

I've spent the last few hours looking at the Gridded Population of the World which consistenly estimates the population density consistent with national censuses and population registers. This would have been a massive job to compile and is really interesting to look at. You can immediately see a strip through the north of India, Pakistan and Bangladesh that is incredibly dense. The north-east of China and the island of Java in Indonesia are also very dense.

I've been looking for some bluetooth headphones that I can use both on a mobile phone and a computer at the same time. I want something portable enough to take with me, but comfortable enough to wear all day. I want something good for listening to music and podcasts on the go, but also for taking calls and video calls. After some investigation I've settled on the Jabra Elite 85h.

I like to think of myself as an egalitarian, but I know I have implicit bias. I've done some tests on Project Implicit and have roughly the implicit biases you would expect for my demographic. This makes me feel a bit sad, but you can't really control your implicit biases, they're a function of your environment and perception growing up. The key question is given that we have implicit biases how do we act against them?

My bashrc file sources the git prompt helper to show the branch I'm on in the prompt. Unfortunately it's quite old and was pointing to the wrong file, how do I find where it is? dpkg -L git | grep prompt Debian and its derivatives such as Ubuntu you can use apt to manage packages (e.g. apt upgrade, apt install). However apt is just a thin layer over dpkg that does useful things like resolving dependencies and downloading files.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.6 Based on the cost of a long-distance plane ticket, estimate the following quantities: (a) the fuel efficiency of a 747, in passenger miles per gallon or passenger kilometres per liter; and (b) the volume of its fuel tank. Check your estimates against the technical data for a 747. A flight from Melbourne to Las Angeles direct costs about $1,000.

Counting is a strangely powerful tool for enduring through something. Standard advice when you're angry is to count to ten. When stretching counting to a target number helps sustain the stretch longer. A good counting based technique for endurance is box breathing. It involves repeatedly inhaling to a count of 4, holding to a count of 4, exhaling to a count of 4 and holding to a count of 4. This is a technique used by Navy SEALs to induce calm and focus.

A password manager is one of the best ways for the majority of people to keep their logins secure. After using KeePass and its derivatives for years, the Kee Firefox Addon dropped support for Keepass and it's now less convenient to use. After looking at the alternatives I'm going to switch to an online alternative. One of the most frequent ways people get their accounts hacked is by password reuse. Their email and password is revealed in some online breach of a website, and then these credentials can be used on other websites.

My first job in analytics was in large part luck. I had an academic background in Physics and Mathematics, some professional programming experience building applications and self-studied computer science. I searched for "python" jobs, since I liked the language, and applied for a job titled something like "Awk, Bash and Grep". I didn't get that job, but was forwarded on to the data engineering team building bespoke reports. That was at a medium size company called Hitwise that provided digital competitive insights.

Edward Chancellor's Devil Take the Hindmost: A History of Financial Speculation is a history of several market bubbles and crashes. It covers bubbles such as the South Sea Bubble, the 1920s bubble in the US stock market preceding the great depression, the dotcom bubble of the 1990's and Japan in the 1980's. The main lessons I took was if a market sounds too good to be true it probably is, that highly leveraged financial instruments tend to prolong and worsen bubbles and often the people who bear the cost of reckless speculation are different to the people who take and profit from it.

I've recently tried to estimate Australian Deaths using life expectancy. This failed badly and I think the reason is demographics; this article looks more into this. The Australian Bureau of Statistics has population by age, and the Australian Institute of Health and Welfare have Mortality Over Time and Regions (MORT) which summarises the current probability of death by age range. Here is a super summarised version of this data: Age Population Death Rate Population Deaths Fraction of Deaths 0-19 25% 0% 0% 0% 20-39 29% 0% 0% 0% 40-59 25% 0% 0% 0% 60-79 17% 1.

I've estimated Australian oil imports; here I check the data to see how reasonable my estimates are. The overall tree diagram for the estimate is below: graph BT; Import[Oil imports1.3 Million Barrels/Day] ImportL[Oil imports200ML/Day] -- Import Barrel[Size of Barrel160L] --|-1| Import Consumption[Oil consumed L/Day200ML/Day] -- ImportL ImportRatio[Oil Imported / Consumed1] -- ImportL CarConsumption[Oil Consumed by Cars100ML/Day] -- Consumption CarFraction[Oil Consumed in Total / Oil Consumed by Cars2] -- Consumption Cars[Number of Cars20 Million] -- CarConsumption ConsumptionCar[Oil Consumed by Car5L/Day] -- CarConsumption People[Number of People25 Million] -- Cars CarPeople[Number of Cars per Person0.

This is a variation of Sanjoy Mahajan's The Art of Insight Section 1.4 (and problem 1.6) How much oil does Australia import (in Barrels per Day)? As in the text we approach this by estimating car consumption. graph BT; Import[Oil imports Barrel/Day] ImportL[Oil imports L/Day] -- Import Barrel[Size of Barrel L] --|-1| Import Consumption[Oil consumed L/Day] -- ImportL ImportRatio[Oil Imported / Consumed] -- ImportL CarConsumption[Oil Consumed by Cars L/Day] -- Consumption CarFraction[Oil Consumed in Total / Oil Consumed by Cars] -- Consumption Cars[Number of Cars] -- CarConsumption ConsumptionCar[Oil Consumed by Car L/Day] -- CarConsumption People[Number of People] -- Cars CarPeople[Number of Cars per Person] -- Cars To estimate imports we estimate demand, since that is estimable.

My 5 year old OnePlus One's power button has finally worn out, to the point where I can't press it. I panicked when the battery ran out - I was afraid I wouldn't be able to power it back on. However I found a video demonstrating how to turn it on with a power cable and the volume button. Pressing the volume down button when you attach the power cable to a computer puts it into recovery mode and you can boot it from there.

How many people die in Australia each year? The life expectancy in Australia is about 80 years, and the population is 25 million. So each year the number of people that die would be about 25 million divided by 80, which is about 300,000. The actual number of people that died in 2018 is 160,000. This is about half my estimate; what am I doing wrong? One factor is life expectancy is at birth, the longer people live the longer they will be expected to live.

How many babies are born in Australia? Australia has 25 million people. I would estimate the birth rate is 0.8 children per person; I think it's slightly less than one. These children are born across life, which is about 80 years. So a really crude estimate for annual births is 25 million people times (0.8 children per person lifetime) divided by 80 years per lifetime. This is 20 million divided by 80 which is 250 thousand.

Exercise from SICP: Exercise 1.5. Ben Bitdiddle has invented a test to determine whether the interpreter he is faced with is using applicative-order evaluation or normal-order evaluation. He defines the following two procedures. (define (p) (p)) (define (test x y) (if (= x 0) 0 y)) Then he evaluates the expression (test 0 (p)) What behavior will Ben observe with an interpreter that uses applicative-order evaluation? What behavior will he observe with an interpreter that uses normal-order evaluation?

Exercise from SICP: Exercise 1.4. Observe that our model of evaluation allows for combinations whose operators are compound expressions. Use this observation to describe the behavior of the following procedure: (define (a-plus-abs-b a b) ((if (> b 0) + -) a b)) Solution There are two possible branches, if b is positive then we get: ((if (> b 0) + -) a b) ((if #t + -) a b) (+ a b) Wheras if b is non-positive we get

Exercise from SICP: Exercise 1.2. Define a function that takes three numbers as arguments and returns the sum of the two larger numbers. Solution The first thing we need to do is to get the largest two numbers from 3 numbers. We can do this with a conditional statement. (define (sum-square-largest-two a b c) (cond ((and (<= a b) (<= a c)) (sum-of-squares b c)) ((and (<= b a) (<= b c)) (sum-of-squares a c)) ((and (<= c a) (<= c b)) (sum-of-squares a b))))

Exercise from SICP: Exercise 1.2. Translate the following expression into prefix form \[ \frac{5 + 4 + (2 - (3 - (6 + \frac{1}{5})))}{3(6-2)(2-7)} \] Solution One way to do this is to read it from the outside in and translate it into a tree (for example the first thing we extract is the division). graph BT; DIV[`/`] -- ANS TOP[.] -- ANS[.] BOTTOM[.] -- ANS TOPSUM[+] -- TOP S1[5] -- TOP S2[4] -- TOP S3[.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.5 Make a tree diagram for your estimate in Problem 1.3. Do it in three steps: (1) Draw the tree without any leaf estimates, (2) estimate the leaf values, and (3) propagate the leaf values upward to the root. This is referring to the suitcase of money. Step 1: Tree graph LR; VolBankNote[Volume of Bank Note] VolSuitcase[Volume of Suitcase] ValueBankNote[Value of Bank Notes in Suitcase] NumBankNote[Number of Bank Notes in Suitcase] ValueSuitcase[Value of Suitcase] NumBankNote -- ValueSuitcase ValueBankNote -- ValueSuitcase VolBankNote --|-1| NumBankNote VolSuitcase -- NumBankNote SuitWidth[Width of Suitcase] -- VolSuitcase SuitHeight[Height of Suitcase] -- VolSuitcase SuitDepth[Depth of Suitcase] -- VolSuitcase BankHeight[Height of Bank Note] -- VolBankNote BankWidth[Width of Bank Note] -- VolBankNote BankDepth[Thickness of Bank Note] -- VolBankNote Step 2: Annotated Leaves graph LR; VolBankNote[Volume of Bank Note] VolSuitcase[Volume of Suitcase] ValueBankNote[Value of Bank Notes in Suitcase

I used to have a fancy kettle that came with settings for heating water to different temperatures between 80° C and 100° C. However it's really easy to get water at any temperature using an ordinary kettle by mixing with refrigerated water. When you mix together two volumes of water at different temperatures their volumes add and the resulting temperature is a volume weighted average of the temperatures. For example if you take 25mL of water at 10° C and 75mL of water at 40° C you will get 100mL of water at 32.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.4 As a bank robber sitting in the vault planning your getaway, do you fill your suitcase with gold bars or $100 bills? Assume first that how much you can carry is a fixed weight. Then redo your analysis assuming that how much you can carry is a fixed volume. As I estimated in suitcase of money the mass of a paper note is about 1 gram, and the volume is about 1 cm³.

When estimating things it's good to find approximate constants, typically ratios, that are easier to remember than things that vary. The Body Mass Index (BMI) is an example for measuring people. It's relatively easy to measure human height, as a human. For example I'm about 180cm tall; the top of my nose is about 170cm, the bottom of my chin is about 160cm and the bottom of my neck is about 150cm.

How much is one cubic centimetre of gold worth? Esitmating with a ring The plan is to estimate the value of gold with the price of a gold ring. This works well because gold is the most expensive part of the ring, and strongly influences the price of the cheapest rings, but the purity is measured and reported. I've seen 14 carat gold rings for around $300, which I'm going to use as a baseline.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.3 In the movies, and perhaps in reality, cocaine and elections are bought with a suitcase of $100 bills. Estimate the dollar value in such a suitcase. Size of $100 note Let's assume a banknote is about the same thickness as paper; Australian notes are probably a little bit thicker. A 500 page ream of paper is about 5cm tall, so each sheet is about 0.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.2 Estimate the mass of air in your bedroom. Pacing out my bedroom it's roughly 3m on each side, and the ceiling is maybe a meter higher than I am, so say 3m high. So the volume of my bedroom is about (3m)³ or 27m³. The density of air is roughly 1 kg/m³, so the mass of air in my bedroom is around 30 kg.

A language that doesn't affect the way you think about programming, is not worth knowing. Alan Perlis I spend a lot of time programming in Python and SQL, some time in Bash and R (or at least tidyverse), and a little in Java and Javascript/HTML/CSS. This set of tools is actually pretty versatile about getting things done, but is fairly narrow from a programming concept perspective. Once in a while I think it's useful to broaden the programming frame to understand different ways of doing things; even if you still stick to the same few languages.

This is from Sanjoy Mahajan's The Art of Insight Problem 1.1 How heavy is a small moving-box filled with books? Guesstimating weight I've moved small boxes of books a few times, it's light enough for me to carry. It's much heavier than a couple of 2kg bag of onions, but probably more similar to a 20kg bag of pool salt. I'd guess it's in the range 10-20kg, so I'd guess around 15kg.

Radio shows, comedy sketch shows and talk shows have the difficult task of filling air time with less structured content. A technique used in all of these mediums to help fill the gaps is a recurring segment. The Saturday Night Live Weekend Update is an example of this. Using a structured recurring segment with a familiar pattern and style gives a structured environment to be creative in. It's really hard to be creative in a completely unstructured and original way, like Monty Python was, since there are a so many options.

When making changes to code tests are a great way to make sure you haven't inadvertently introduced regressions. This means that you can make changes much faster with more confidence, knowing that your tests will catch many careless mistakes. But what do you do when you're working with a legacy codebase that doesn't have any tests? One method is creating diff tests; testing how your changes impact the output. For batch model training or ETL pipeline there's typically a natural way to do this.

When making changes to a new model training pipeline I find it really useful to understand the dataflow. Analytics workflows are done as a series of transformations, taking some inputs and producing some outputs (or in the case of mutation; an input is also an output). Seeing this dataflow helps give a big picture overview of what is happening and makes it easier to understand the impact of changes. Generally you can view the process as a directed and (hopefully) acyclic graph.

A lot of analytics code I've read is a very long procedural chain. These can be hard to follow because the only way to really know what's going on in any point is to insert a probe to inspect the inputs and outputs at that stage. Breaking these into functions is a really useful way of making the code easier to understand, change and find bugs in. In Martin Fowler's Refactoring he mentions that whenever there's a block of code that has (or requires) a comment to describe what it does, that's a good opportunity to package that code into a function.

I love having a clean desk and empty inbox. But I hate spending the time cleaning my desk and processing emails. It feels like wasted time where I could do something better. However having "tidy time" to maintain things is important. A while ago I read David Allen's Getting Things Done. When I tried to implement it I got stuck on the notion of a weekly review. Setting aside some time every week to see how you're progressing on tasks and to process any new tasks.

Waiting for independent I/O can be a performance bottleneck. This can be things like downloading files, making API calls or running SQL queries. I've already talked about how to speed this up with multiprocessing. However it's easy to move to the more recent concurrent.futures library which allows running on threads as well as processes, and allows handling more complicated asynchronous flows. From the previous post suppose we have this multiprocessing code:

You've done an analysis and generated an output file in a Jupyter notebook. How do you get it down to your computer? For a local server you could find it in your filesystem, or for a remote server copy it with something like scp. But there are easier ways. You can download individual files from the file navigator (which you can get to by clicking on the Jupyter icon in the top left corner).

Pretty frequently I start writing some code, when I realise there's another change I need to make before I can continue. I like to make lots of small atomic changes to a code base because it lets me test more quickly and catch errors earlier. I used to do this by saving my changes in a temporary file, but this was clunky. A better way is with git stash. But git stash reverts all files; and very often I want to keep some, especially configuration parameters.

Logarithms are a handy calculational tool for turning multiplication into addition. You don't need any special tools to calculate approximate logarithms by hand, you just need to remember a few identities. An easy example of how this works is with the identity \( 2^{10} = 1024 \approx 1000 = 10^3 \). Using base 10 logarithms this means that \( 10 \log 2 \approx 3 \), or \( \log 2 \approx 0.

A good technique for deeply understanding something is to try to solve it yourself first. Sometimes this can even lead to better methods or new discoveries. I heard an interesting technique from Jeremy Howard in one of the fast.ai courses about how to read a paper. First read the abstract and introduction. Then spend a couple of days trying to implement what you think they're talking about. Then go back and read the rest of the paper and see how it compares to what you did.

The state government of Victoria, Australia has recently announced a plan on how to respond to the current Covid-19 pandemic. Based on epidemiological modelling they have set to reduce restrictions based on 14 day averages of new case numbers. If the 14 day average daily new cases are 30-50 in 3 weeks they will reduce restrictions; if they are below 5 a month after that they will reduce restrictions again.

It's worthwhile spending some time thinking about how you spend your time. Time and energy are among your most valuable resources. A regular investment of time can build into substantial assets, but if you don't budget time it's easily misspent. I don't believe that you should allocate away all of your time, but setting some time constraints is important. If you don't put the big rocks of things that are important to you first in the jar first, all the sand and water of mundane things will fill it up.

Today when I tried to connect to VcXsrv after running it with XLaunch it didn't work. I'd had it working for months and so was surprised it suddenly stopped working. The reason was simple; the IP subnet WSL2 had changed and so it was now being blocked by a firewall. Annoyingly there is very little feedback as to why it can't connect to an XServer. I went back through my previous instructions of setting up an X server in WSL2, but noticed something.

Today I saw a picture in someone's windows "Always Exceed Everyone's Expectations". My initial reaction was that was a quick way to burnout - trying to always exceed expectations sounds like running on a treadmill that gets faster and faster. But another way to look at it is to set lower expectations and only commit when you can confidently deliver. In another expression "under-promise and over-deliver". Consistently delivering what you promise to customers is the way to build trust and loyalty.

I've been surprised to learn that financial bubbles and collapses are actually hundreds of years old. I learned this reading the book Devil Takes the Hindmost: A History of Financial Speculation by Edward Chancellor. The chapters on the South Sea Bubble and the following craze over investing in South America sound thoroughly modern; except they happened 200-300 years ago. In fact Isaac Newton lost £20,000 by investing in the bubble. Chancellor describes futures, options, and margin loans - things I had wrongly assumed were more modern inventions.

Categorical objects with a large number of categories are quite problematic for modelling. While many models can work with them it's really hard to learn parameters across many categories without doing a lot of work to get extra features. If you've got a related dataset containing these categories you may be able to meaningfully embed them in a low dimensional vector space which many models can handle. Categorical objects occur all the time in business settings; products, customers, groupings and of course words.

I've had a few discussions with people, especially analysts, about how to learn programming. Generally I encourage them to find a project they want to accomplish and try to learn programming on the way. However I really struggle to find resources to recommend because they tend to spend a lot of time teaching programming concepts from scratch. I wonder if a better way to teach these things would be to start with code that's close to what they want to accomplish, and get them to edit it.

A while ago I came across Cynthia Rudin through their work on the FICO Explainable Machine Learning Challenge. Her team got an honourable mention and she wrote an opinion with Joanna Radin on explainable models. I think the article was hyperbolic on claiming interpretable models always work as well as black box models. On the other hand I only came across her because of this article, so taking an extreme viewpoint in the media is a good way to get attention.

Sometimes you want to classify documents, but you don't have an existing classification. Building a classification that is mutually exclusive and completely exhaustive is actually very hard. Topic modelling is a great way to quickly get started with a basic classification. Creating a classification may sound easy until you try to do it. Think about novels; is a Sherlock Holmes novel a mystery novel or a crime novel (or both)? Or do we go more granular and call it a detective novel, or even more specifically a whodunit?

When doing a left join in SQL any filtering of the table after the join will turn it into an inner join. However there are some easy ways to do the filtering first. Suppose you've got some tables related to a website. The pages table describes the different pages on the site. page_id page_name 1 home 2 checkout 3 terms Another pageviews describes the daily activity:

A coarse geocoder takes a human description of a large area like a city, area or country and returns the details of that location. I've been looking into the source of the excellent Placeholder (a component of the Pelias geocoder) to understand how this works. The overall approach is straightforward, but it takes a lot of work to get it to be reliable. A key component geocoder is a gazetteer that contains the names of locations.

A lot of information is embedded in HTML pages, which contain both human text and markup. If you ever want to extract this information, don't use regex use a parser. Python has an inbuilt library html.parser library to do just that. The excellent html2text library uses it to parse HTML into markdown, which you can use for removing formatting. However for your own purposes you can use a similar approach to build a custom parser by subclassing HTMLParser.

Placeholder is a great library for Coarse Geocoding, and I'm using it for finding locations in Australia. In my application I want to get the location to a similar level of granularity; however the input may be for a higher level of granularity. Placeholder doesn't directly provide a method to do this, but you can use their SQLite database to do it. For example to find the largest locality for East Gippsland, with Who's On First id 102049039, you can use the SQL.

A monad in languages like Haskell is used as a particular way to raise the domain of a function beyond where it was domain. You can think of them as a generalised form of function composition; they are a way of taking one type of function and getting another function. A very useful case is the maybe monad used for dealing with missing data. Suppose you've got some useful function that parses a date: parse_date('2020-08-22') == datetime(2020,8,22).

If you've got a distribution you may want a way to tell if it has multiple components. For example a sample of heights may have a couple of peaks for different gender, or other attributes. While you could determine this through explicitly modelling them as a mixture the results are sensitive to your choice of model. Another approach is statistical tests for multimodality. One common test is Silverman's Test which checks for the number of modes in a kernel density estimate; the trick is choosing the right width.

There are always more things you can be doing than the time you have. If you try to do everything you will end up accomplishing nothing. It can be hard to say "no" to someone, but it's important in order to focus on your priorities. I find it useful to have priorities and goals to set a direction. Often the actual choice of goals doesn't matter as much as that I make them, and regularly review them.

Sometimes you may want to experiment with sessions and need to hand-roll your own in SQL. There's a good mode blog on how to do this. If you're using Postgres or Greenplum you may be able to use Apache Madlib's Sessionize for the basic case. This blog post will give a very brief summary of how to do this with some examples in Presto/Athena. The idea of a session is to capture a continuous unit of user activity.

When creating a table in Athena I got the error: Invalid column type for column: Unsupported Hive type: timestamp with time zone. Unfortunately it can't support timestamps with timezone. In my case all the data was in UTC so I just needed to remove the timezone to create the table. The easiest way to do that was to cast it to a timestamp (without a timezone). cast(event_time as timestamp)

Python is a very versatile multiparadigm language with a great ecosystem of libraries. However it is not a functional programming lanugage, as I know some people have described it. While you can write it in a functional style it goes against common practice, and has some practical issues. There is no fundamental definition of a functional programming language but two core concepts are that data are immutable and the existence of higher order functions.

Differentiation is the process of creating a local linear approximation of a function. This is useful because linear functions are very well understood and efficient to work with. One application of them is gradient descent, often used for fitting models in machine learning. In this context a function is something that maps between coordinate spaces. For example consider an image classifier that takes a 128x128 pixel image with three channels for colours (Red, Green, Blue) and returns a probability that the image contains a cat and the probability that the image contains a dog.

Groups can be thought of a mathematical realisation of symmetry. For example the symmetric groups are all possible permutations of n elements. Or the dihedral groups are the symmetries of a regular polygon. A questions mathematicians ask is what kinds of groups are there? One way to tackle this is to try to decompose them. One way of doing this is a decomposition series of normal subgroups. \[ 1 = H_0\triangleleft H_1\triangleleft \cdots \triangleleft H_n = G \]

A challenge of data analytics is that the data can change as well as the code. The systems producing and collecting data are often changed and can lead to missing or corrupt data. These can easily corrupt reports and machine learning systems. Worst of all the data may be lost permanently. So if you're going to use some data it's important to check the data regularly to catch the worst kind of mistakes as early as possible.

You don't need a lot of data to prove a point. People often think statistics requires big expensive datasets that cost a lot to acquire. However in relatively unexplored spaces a small amount of data can have high yield in changing a decision. I've been working on some problems around web sessionisation. The underlying model is that when someone visits your website they may come at different times for different reasons.

Even when there's a specific field for a price there's a surprising number of ways people write it. This is what the tool price-parser solves. Unfortunately it doesn't work too well on salaries, which tend to be ranges and much higher, but the approach works. Price parser has a very large set of tests covering different ways people write prices. The solution is a simple process involving a basic regular expression, but it solves all these different cases.

People write locations in many different ways. This makes them really hard to analyse, so we need a way to normalise them. I've already discussed how Placeholder is useful for coarse geocoding. Now I'm trying to apply it to normalising locations from Australian Job Ads in Common Crawl. The best practices when using Placeholder are: Go from the most specific location information (e.g. street address) to the most general (e.

I've been thinking about how to convert HTML to Text for NLP. We want to at least extract the text, but if we can preserve some of the formatting it can make it easier to extract information down the line. Unfortunately it's a little tricky to get the segmentation right. The standard answers on Stack Overflow are to use Beautiful Soup's getText method. Unfortunately this just turns every tag into the argument, whether it is block level or inline.

I'm trying to find a way of converting HTML to something meaningful for NLP. The html2text library converts HTML to markdown, which strips away a lot of the meaningless markup. I've already resolved an issue with multiple types of emphasis. However HTML in the wild has all sort of weird edge cases that the library has trouble with. In this case I found a term that was emphasised twice: <strong><strong>word</strong></strong>. I'm pretty sure for a browser this is just the same as doing it once; <strong>word</strong>.

I've been trying to find a way of converting HTML to something meaningful for NLP. The html2text library converts HTML to markdown, which strips away a lot of the meaningless markup. But I quickly hit an edge case where it fails, because parsing HTML is surprisingly difficult. I was parsing some HTML that looked like this: Some text.<br /><i><b>Title</b></i><br />... When I ran html2text it produced an output like this:

The simplest way to model probability of a system is through symmetry. For example the concept of a "fair" coin means there are two possible outcomes that are indistinguishable. Because each result is equally likely the outcome is 50/50 heads or tails. Similarly for a fair die there are 6 possible outcomes, that are all equally likely. This means they each have the probability 1/6. The idea of symmetry is behind random sampling.

Today I went through the painful exercise of culling my notebooks. My honours notebooks, independent research and work from textbooks and courses. These are things I spent a large part of my early life and energy on. Even though I haven't looked at them for years they are very hard to let go. A large amount of the material is pure mathematics. Notes on differential geometry, topology, and measure theory. These are particularly vexing because I don't believe they hold much real value.

It can be convenient to directly publish a mixture of prose, source code and graphs. It ensures the published code actually runs and makes it much easier to rerun at a later point. I’ve done this before in Hugo with R Blogdown, and now I’m experimenting with Jupyter notebooks. The best available option seems to be nb2hugo which converts the notebook to markdown, keeping the front matter exporting the images.

There's a lot of great freely available resources in NLP right now; and the field is moving quickly with the recent success of neural models. I wanted to mention a few that look interesting to me. Jurefsky and Martin's Speech and Language Processing The third edition is a free ebook that is in progress that covers a lot of the basic ideas in NLP. It's got a great reputation in the NLP community and is nearly complete now.

I often find listening more productive than talking, but still find it easy to spend a lot of meetings talking. When I get curious I ask lots of questions in a meeting that can take it off on a tangent, especially switching from high level to detail. If you find yourself in a similar situation give yourself a small speaking quota. I got the idea from a former management consultant, who when he was a junior was told he was only allowed to say one thing in a meeting.

I'm sitting in a meeting listening to an update. They've missed the point, and they're focusing on the wrong thing. I start to get frustrated; why are they so far off track? Why haven't they taken the time to understand the problem? This isn't a helpful reaction; getting short tempered won't help resolve the problem. I haven't taken the time to understand the speaker and their perspective. Why do they think this is the right thing to focus on?

In the past two years the best performing NLP models have been based on transformer models trained on an enormous corpus of text. By understanding how language in general works they are much more effective at detecting sentiment, classifying documents, answering questions and translating documents. However in any particular case we are solving a particular task in a certain domain. Can we get a better performing model by further training the lanugage model on the specific domain or task?

How can we evaluate how good a machine generated translation is? We could get bilingual readers to score the translation, and average their scores. However this is expensive and time consuming. This means evaluation becomes a bottleneck for experimentation If we need hours of human time to evaluate an experiment this becomes a bottleneck for experimentation. This motivates automatic metrics for evaluation machine translation. One of the oldest examples is the BiLingual Evaluation Understudy (BLEU).

I've recently started working with WSL2 on my Windows machine, but have had trouble getting an X server to run. This is an issue for me because running Emacs with Evil keybindings under Windows Terminal I often find there's a lag in registering pressing escape which leads to some confusing issues (but vanilla Vim is fine). But having an X Server would also allows running any Linux graphical application under X.

Tracking a metric can help to drive dramatic improvements. When your team is focused on a metric you can test what has impact and quickly optimise it. However for this to work it's important to be something you can actually impact. When people start looking for a metric to track they want to look for things that have a direct impact on the business, such as revenue, share price or customer satisfaction.

I keep my personal configuration files in a public dotfiles repository. This means that whenever I'm on a new machine it's very easy to get comfortable in a new environment. However I find I often need machine specific configuration, so I provide ways to override them with local configuration. When I get to a new machine I'll pretty quickly want some of my usual configuration (although I don't need it). I can clone or download a zipfile of my dotfiles and then install it via some symlinks via a bootstrap bash script.

Sometimes you want a different git configuration in different contexts. For example you might want different author information, or to exclude files for only some kinds of projects, or to have a specific template for certain kinds of projects. The easiest way to do this consistently is with a includeIf statement. For example to have custom options for any git repository under a folder called apache add this to the bottom of your ~/.

Python concurrent.futures are a handy way of dealing with asynchronous execution. However if you're not careful it will swallow your exceptions leading to difficult to debug errors. While you can perform concurrent downloads with multiprocessing it means starting up multiple processes and sending data between them as pickles. One problem with this is that you can't pickle some kinds of objects and often have to refactor your code to use multiprocessing.

My friend has four different magnets for plumbers on his fridge. Three of them are generic rectangular magnets that have generic information and contact details. One of them was in the shape of a dripping tap, mentioning they were experts in leaks and drips. If they had a leaking faucet it's pretty easy to guess which plumber they would call; the specialists in dripping taps. On the other hand if they had a clogged toilet it's down to chance which of the plumbers they would call, although they're less likely to call the dripping tap specialist they're also more likely to forget to look at the fridge and just search for a plumber online.

In the 90s Microsoft famously used a strategy of embracing other protocols, then adding extensions to their implementation until it's no longer compatible and utilising their market leverage to extinguish competing implementations. While "EEE" is normally associated with Microsoft many of the software titans use it as an effective strategy to further their existing dominance into new markets. Embracing a technology with an existing market is an effective way to quickly gain adoption.

The University of Adelaide, being a sandstone Group of Eight University, has the archaic ceremony of a mace-bearer leading the processing carrying a heavy piece of expensive metal. When I graduated with my Bachelor of Science I was fortunate enough to be that mace-bearer. Unfortunately I wasn't really prepared for the formality. The ceremony was on a typical Adelaide summer's day, hot and dry. I was going out to lunch with my parents afterwards, so I wanted to make sure I was comfortable.

When I don't have clear goals, I'm more likely to spread myself too thin. It's easy to get in a reactive mode, where I say "yes" to anything that comes on my plate. But that means I don't have the time to focus on the important things and am jumping from task to task. It takes time to set goals and priorities, but when I don't take that time I end up spending much more time on busywork.

Information is useful in that it helps make better decisions. This is much easier if the data is represented in a way that closely match the conceptual model of the business. Building a useful view of the data can dramatically decrease the time and cost of answering questions and even elevate the conversation to answering deeper questions about the business. A typical example of where analysis can help is trying to increase revenue of a digitally sold product.

For any actionable item there are four ways to handle it: do it, defer it, delegate it or delete it. Delegation is an often overlooked powerful option to handle things. It's not just for high powered executives to delegate down to their personal assistants; even if you don't have any reports it's possible to delegate. You can delegate in three directions: down, sideways and up. Downwards delegation is the classic kind that comes to most people's minds.

When training machine learning models typically you get a training dataset for fitting the model and a test dataset for evaluating the model (on small datasets techniques like cross-validation are common). You typically assume the performance on your chosen metric on the test dataset is the best way of judging the model. However it's really easy for systematic biases or leakage to creep into the datasets, meaning that your evaluation will differ significantly to real world usage.

Deep Neural Networks have transformed dealing with unstructured data like images and text, making totally new things possible. However they are difficult to train, require a large amount of relevant training data, are hard to interpret, hard to debug and hard to refine. I think for these reasons there's a lot of space to use neural networks as a building block for extracting structured data for less parameterised models. Josh Tenenbaum gave an excellent keynote at ACL 2020 titled Cognitive and computational building blocks for more human-like language in machines.

I remember three things from lectures in my first year of university. One is a chemistry professor acting out the three vibrational modes of water; his head being the Oxygen atom and his hands being the Hydrogen atoms. Another is Rod Crewther demonstrating torque by showing how difficult it to open a heavy lecture door by pushing at the hinge. The third is how Nick Buchdahl illustrated the mean value theorem.

The traditional way to train an NER model on a new domain is to annotate a whole bunch of data. Techniques like active learning can speed this up, but especially neural models with random weights require a ton of data. A more modern approach is to take a large pretrained NER model and fine tune it on your dataset. This is the approach of AdaptaBERT (paper), using BERT. However this takes a large amount of GPU compute and finicky regularisation techniques to get right.

Dependency trees are a remarkably powerful tool for information extraction. Neural based taggers are very good and Universal Dependencies means the approach can be used for almost any language (although the rules are language specific). However syntax can get really strange requiring increasingly complex rules to extract information. The pyBART system solves this by rewriting the rules to be half a step closer to semantics than syntax. I've seen that dependency based rules are useful for extracting skills from noun phrases and adpositions.

Many documents available on the web have meaningful markup. Headers, paragraph breaks, links, emphasis and lists all change the meaning of the text. A common way to deal with HTML documents in NLP is to strip away all the markup, e.g. with Beautiful Soup's .get_text. This is fine for a bag of words approach, but for more structured text extraction or language model this seems like throwing away a lot of information.

Modern Javascript web frameworks often embed the data used to render each webpage in the HTML. This means an easy way of extracting data is capturing the string representation of the object with a pushdown automoton and then parsing it. Python's inbuilt json.loads is effective, but won't handle very dynamic Javascript, but demjson will (another, much faster alternative is Chompjs. The problem shows up when using json.loads as the following obscure error:

Beautiful soup can be a useful library for extracting information from HTML. Unfortunately there's a lot of little issues I hit working with it to extract data from a careers webpage using Common Crawl. The library is still useful enough to work with; but the issues make me want to look at alternatives like lxml (via html5-parser). The source data can be obtained at the end of the article. Use a good HTML parser Python has an inbuilt html.

When fetching large amounts of data from the internet a best practice is caching all the data. While it might seem easy to extract just the information you need, it's easy to hit edge cases or changing structure, and you can never use the data you throw away. This is easy to do in the Web ARChive (WARC) format with warcio used by the Internet Archive and Common Crawl. from warcio.

It's very useful to diverge on ideas before converging on a solution. Trying to do both at the same time tends to stifle creativity and lead to less innovative solutions. I find the creative process of brainstorming is more effective if I do it separately to refining ideas. Taking the time to brainstorm leads to better solutions, whether thinking about what to work on, planning out a presentation or designing a technical solution.

Downloading files can often be a bottleneck in a data pipeline because network I/O is slow. A really simple way to handle this is to run multiple downloads in parallel accross threads. While it's possible to deal with the unused CPU cycles using asynchronous processing, in Python it's generally easier to throw more threads at it. Using multiprocessing can be very simple if you can turn make the processing occur in a pure function or object method, and both the variables are results are picklable.

Data pipelines can often be thought of as a chain of pure functions passing data between them, even if they are not implemented that way. As long as you can access the sources you can always rerun the pipeline from end to end. However if the processing takes a long time to run it can be convenient to cache some step as a sink, especially if steps are likely to fail.

I am trying to extract Australian Job Postings from Web Data Commons which extracts structured data from Common Crawl. I previously came up with a SPARQL query to extract the Australian jobs from the domain, country and currency. Unfortunately it's quite slow, but we can speed it up dramatically by replacing it with a similar script in grep. With a short grep script we can get twenty thousand Australian Job Postings with metadata from 16 million lines of compressed nquad in 30 seconds on my laptop.

Sometimes you have some description of a location and want to work out where it is. This is called geocoding; if you just want to know what state or country it's in it's called coarse geocoding. I found that while many structured JobPostings contain a country some have it as a description rather than a country code, and some put the location in other fields. We can often find the country using geocoding.

I am trying to understand how the JobPosting schema is used in Web Data Commons structured data extracts from Common Crawl. I wrote a lot of ad hoc Python to get usage statistics on JobPosting. However SPARQL is a tool that makes it much easier to answer these kinds of questions. After reading in the graphs individually they can be combined into a rdflib.Dataset so we can query them all together.

I've been using RDFLib to parse Job posts extracted from Common Crawl. RDF Literals It automatically parses XML Schema Datatypes into Python datastructures, but doesn't handle the <http://schema.org/Date> datatype that commonly occurs in JSON-LD. It's easy to add with the rdflib.term.bind command, but this kind of global binding could lead to problems. When RDFLib parses a literal it will create a rdflib.term.Literal object and the value field will contain the Python type if it can be successfully converted, otherwise it will be None.

A job posting has a description, a company, sometimes a salary, ... and what else? Schema.org have a detailed JobPosting schema, but it's not immediately obvious what is important and how to use it. However the Web Data Commons have extracted JobPostings from hundreds of thousands of webpages from Common Crawl. By parsing the data we can see how these are actually used in practice which will help show what is actually useful in describing a job posting.

The Web Data Commons has a vast repository of structured RDF Data about local businesses, hostels, job postings, products and many other things from the internet. Unfortunately it's not in a format that's easy to do analysis on. We can stream the nquad format to get RDFlib Graphs, but we still need to convert the data into a form we can do analysis on. We'll do this by turning the relations into dictionaries of properties to the list of objects they contain.

The Web Data Commons extracts structured RDF Data from about one monthly Common Crawl per year. These contain a vast amount of structured information about local businesses, hostels, job postings, products and many other things from the internet. Python's RDFLib can read the n-quad format the data is stored in, but by default requires reading all of the millions to billions of relations into memory. However it's possible to process this data in a streaming fashion allowing it to be processed much faster.

I've been using Github Actions to publish this website for almost a month. The experience has been great; whenever I push a commit it gets consistently published without me thinking about it within minutes. However I have one concern; I'm passing my rsync credentials into an external action. I've specified a tag in my yaml uses: wei/rclone@v1, but it would be easy for the author to move this tag to another commit that sends my private credentials to their personal server.

When checking my work in SQL one of the first things I do is confirm a column I expect to be unique is. Many tables have a unique key at the level they are at; for session level data it's a session id, for user level data it's a user_id or for daily data it's a date. It's generally a useful thing to check because all it takes is one bad join to end up with a bunch of duplicate (or dropped) rows.

One of the most important abilities of an analyst is to be able to check your work. It's really easy to get incorrect data, have issues in data processing, or even misunderstand what the output means. But if your work is valuable enough to change a decision it's worth doing whatever you can to check it's right. When you get to the end of a long analysis it seems like a time to relax and be glad the hard work is over.

Sometimes you may have to parse a string with backslash escapes; for example "this is a \"string\"". This is quite straightforward to parse with a state machine. The idea of a state machine is that the action we need to take will change depending on what we have already consumed. This can be used for proper regular expressions (without special things like lookahead), and the ANTLR4 parser generator can maintain a stack of "modes" that can be used similarly.

I've been using data from the Adzuna Job Salary Predictions Kaggle Competition to extract skills, find near duplicate job ads and understand seniority of job titles. But the dataset has heavily processed ad text which makes it harder to do natural language processing on. Instead I'm going to find job ads in Common Crawl's, a dataset containing over a billion webpages each month. The Common Crawl data is much better because it's longitudinal over several years, international, broad and continually being updated.

Common Crawl builds an open dataset containing over 100 billion unique items downloaded from the internet. There are petabytes of data archived so directly searching through them is very expensive and slow. To search for pages that have been archived within a domain (for example all pages from wikipedia.com) you can search the Capture Index. But this doesn't help if you want to search for paths archived across domains. For example you might want to find how many domains been archived, or the distribution of languages of archived pages, or find pages offered in multiple languages to build a corpus of parallel texts for a machine translation model.

Common Crawl builds an open dataset containing over 100 billion unique items downloaded from the internet. You can search the index to find where pages from a particular website are archived, but you still need a way to access the data. Common Crawl provides the data in 3 formats: If you just need the text of the internet use the WET files If you just need the response metadata, HTML head information or links in the webpage use the WAT files If you need the whole HTML (with all the metadata) then use the full WARC files The index only contains locations for the WARC files, the WET and WAT files are just summarisations of it.

Common Crawl builds an open dataset containing over 100 billion unique items downloaded from the internet. Every month they use Apache Nutch to follow links accross the web and download over a billion unique items to Amazon S3, and have data back to 2008. This is like what Google and Bing do to build their search engines, the difference being that Common Crawl provides their data to the world for free.

Different industries have different ways of distinguishing seniority in a job title. Is a HR Officer more senior than a HR Administrator? Is a PHP web developer more skilled than a PHP developer? How different is a medical sales executive to general sales roles? Using the jobs from Adzuna Job Salary Predictions Kaggle Competition I've found common job titles and can use the advertised salary to help understand them. Note that since the data is from the UK from several years ago a lot of the details aren't really applicable, but the techniques are.

A job ad title can contain a lot of things like location, skills or benefits. I want a list of just the job titles, without the rest of those things. This is a key piece of information extraction that can be used to better understand jobs, and built on by understanding how different job titles relate, for example with salary. To do this we first normalise the words in the ad title, doing things like removing plurals and expanding acronyms.

When evaluating whether to use an open source project I generally want to know how active the project is. A project doesn't need to be active to be usable; mature and stable projects don't need to change much to be reliable. But if a project has problems or missing essential features, or is in an evolving ecosystem (like any web project or kernel drivers), it's important to know how fast it changes.

Trying to normalise text in job titles I need a way to convert plural words into their singular form. For example a job for "nurses" is about a "nurse", a job for "salespeople" is about a "salesperson", a job for "workmen" is about a "workman" and a job about "midwives" is about a "midwife". I developed an algorithm that works well enough for converting plural words to singular without changing singular words in the text like "sous chef", "business" or "gas".

When examining words in job titles I noticed that if was common to see titles written as "head of ..." or "director of ...". This is unusual because most role titles go from specific to general (e.g. finance director) to you look backwards from the role word. In the "A of B" format the role goes from specific to general and so you have to reverse the search order. One solution is to rewrite "director of finance" to "finance director".

A friend needed to generate a hundred contracts and their HR information system wasn't working properly. I helped them implement a workaround solution by using mail merge to generate a PDF for every contract, which saved them a lot of time filling in the details of each contract. I couldn't automatically generate the PDF despite some efforts, but using mail merge was much quicker and more reliable than filling in all the contract details manually into the template.

Sometimes you have a long sequence you want to break into smaller sized chunks. This is generally because you want to use some downstream process that can only handle so much data at a time. This is common in stochastic gradient descent in deep learning where you are constrained by the memory on the GPU. But this is also useful for API calls that can take a list, but can't handle all the data at once.

I have a tendency to create really complex metrics. Sometimes when I'm analysing data I'll need to transform the data to understand it. I often calculate the ratio of common metrics to get a more stable rate. Or when building a machine learning model I'll find that log-loss or root mean square log error is the right metric. This can be appropriate for gaining insight or training a model, but it's not good for communication.

I've been trying to find near duplicate job ads in the Adzuna Job Salary Predictions Kaggle Competition. Job ads can be duplicated because a hirer posts the same ad multiple times to a job board, or to multiple job boards. Finding exact duplicates is easy by sorting the job ads or a hash of them. But the job board may mangle the text in some way, or add its own footer, or the hirer might change a word or two in different posts.

We've found pairs of near duplicate texts in 400,000 job ads from the Adzuna Job Salary Predictions Kaggle Competition. When we tried to extracted groups of similar ads by finding connected components in the graph of similar ads. Unfortunately with a low threshold of similarity we ended up with a chain of ads that were each similar, but the first and last ad were totally unrelated. One way to work around this is to find cliques, or a group of job ad were every job ad is similar to all of the others.

My first professional job was for Haese mathematics which is a small family-owned South Australian business that writes and publishes mathematics textbooks. Working for a small company was a really interesting experience, I learned software development for their applications both for students and teachers, made animations, edited audio and did layout and graphic design of the books. Unfortunately I didn't make the effort to learn much about the business itself, which makes me wonder how big the market is for mathematics textbooks.

Placing options on a scatterplot of costs versus benefits is a common practice for prioritising opportunities and solutions. The primary benefit of this approach is it can spark discussions. When people see the options on the canvas they it can help uncover unseen issues and opportunities. Getting a group of people involved in putting it together can help get them on the same page. The primary risk of this approach is getting too precise about it.

When you have behavioural data between actors and events you naturally get a bipartite graph. For example you can have the actors as customers and events as products that are purchased, or the actors as users of a website and the events as videos that are viewed, or the actors as members of a forum and the events as posts they comment on. One of the ways to represent this is to relate actors by the number of events they both participate in.

Suppose you're running a website with tens of thousands of different products, and no satisfactory way to group them up. Even a mediocre clustering can really help bootstrap your understanding. You can use the clusters to see new patterns in the data, and you can manually refine the clusters much more easily than you can make them. There are many techniques to cluster structured data or even detect them as communities in the graph of interactions with your users.

Today I was picking grapes from their vine for my partner's grandmother. They had been left too long and many were rotting or had bright blue spots where some form of fungus or algae was growing on them. I sorted the grapes into piles of rotten grapes and edible grapes. When I picked a big bunch of grapes with a couple of rotting overripe grapes I sorted it into the rotten pile, despite there being a dozen ripe looking grapes.

I really like the idea of having a process triggered automatically when I push code. Github actions gives a way to do this with Github repositories, and this article was first published with a Github action. While convenient for simple things Github actions seem hard to customise, heavyweight to configure and give me security concerns. My workflow for publishing this website used to be commit and push the changes and run a deploy script.

Estimating projects is notoriously difficult, and the larger the project the harder to estimate. But even small pieces of work for a single person are easy to underestimate. When you make an estimate base it on actual elapsed times of similar projects, always try to overestimate the time, and reduce the scope before promising more than you can deliver. Everyone knows that construction jobs are typically going to take longer and cost more than quoted, from home renovations to major construction projects.

People using a website or app will have different patterns of behaviours. It can be useful to cluster the customers or products to help understand the business and make better strategic decisions. One way to view this data is as an interaction graph between people and the product they interact with. Clustering a graph of interactions is called "community detection" Santo Fortunato's review article and user guide provides a really good introduction to community detection.

I was trying to load a local file in a HTML page and got a Cross-Origin Request Blocked error in my browser. The solution was to start a Python web server with python3 -m http.server. I had a JSON file I wanted to load into Javascript in a HTML page. Looking at StackOverflow I found I found fetch could do this fetch("test.json") .then(response => response.json()) .then(json => process(json)) Where process is some function that acts on the data; console.

When I'm in a comfortable environment I love to talk. This can be really useful for working through a problem by bouncing ideas off of other people, or for educating people and getting a point accross. But in getting something done I find listening is much more powerful than talking. There's lots of reasons to spend more time listening than talking. When you get a greater diversity of ideas you generally get to a better solution, and often the quieter people in the room have a valuable perspective.

I've found pairs of near duplicates texts in the Adzuna Job Salary Predictions Kaggle Competition using Minhash. One thing that would be useful to know is what the common sections of the ads are. Typically if they have a high 3-Jaccard similarity it's because they have some text in common. The most asymptotically efficient to find the longest common substring would be to build a suffix tree, but for experimentation the heuristics in Python's DiffLib work well enough.

My first instinct when dealing with a new problem is to try to find a complex technique to solve it. However I've almost always found it more useful to start with a simple model before trying something more complex. You gain a lot from trying simple models and the cost is low. Even if they're not enough to solve the problem (which they can be) they will often give a lot of information about the problem which will set you up for later techniques.

You've got a couple of groups and you want to get every possible combination of them. This is called the Cartesian Product of the groups. There are standard ways of doing this in R and Python. Python: List Comprehensions Concretely we've got (in Python notation) the vectors x = [1, 2, 3] and y = [4, 5] and we want to get all possible pairs: [(1, 4), (2, 4), (3, 4), (1, 5), (2, 5), (3, 5)]`.

The Beta Function comes up in the likelihood of the binomial distribution. Understanding its properties is useful for understanding the binomial distribution. The beta function is given by \( B(a, b) = \int_0^1 p^{a-1}(1-p)^{b-1} \rm{d}p \) for a and b positive. If you have \(N\) flips of a coin of which \(k\) turn heads the likelihood is proportional to \( p^{k}(1-p)^{N-k} \) for the probability p between 0 and 1. So the beta function can be seen as the normaliser of the likelihood, with \( a = k + 1 \) and \( b = N - k + 1 \) (or inversely \( k = a - 1 \) and \( N = a + b - 2 \)).

Suppose that you flip a fair coin 10 times, how many heads will you get? You'd think it was close to 5, but it might be a bit higher or lower. If you only got 7 heads would you reconsider you assumption the coin is fair? What if you got 70 heads out of 100 flips? This might seem a bit abstract, but the inverse problem is often very important. Given that 7 out of 10 people convert on a new call to action, can we say it's more successful than the existing one that converts at 50%?

We've found pairs of near duplicates texts in the Adzuna Job Salary Predictions Kaggle Competition using Minhash. But many pairs will be part of the same group, in an extreme case there could be a group of 5 job ads with identical texts which produces 10 pairs. Both for interpretability and usability it makes sense to extract these groups from the pairs. Extracting the Groups Directly with Union Find Each band of the LSH consists of buckets of items that may be similar; you could view the buckets as a partition of the corpus of all documents.

I'm trying to find near duplicates texts in the Adzuna Job Salary Predictions Kaggle Competition. In the last article I built a collection of MinHashes of the 400,000 job ads in half an hour in a 200MB file. Now I need to efficiently search through these minhashes to find the near duplicates because brute force search through them would take a couple of days on my laptop. MinHash was designed to approach this problem as outlined in the original paper.

I wanted to work through an example of applying Bayes rule to update model paremeters based on toy data This example comes from Kruschke’s Doing Bayesian Data Analysis, Section 5.3. The model is that we have a coin and we’re trying to estimate the bias in the coin, that is the probability that it will come up heads when flipped. For simplicity we assume the bias, theta is a multiple of 0.

I'm trying to find near duplicates texts in the Adzuna Job Salary Predictions Kaggle Competition. I've found that that the Jaccard index on n-grams is effective for finding these. Unfortunately it would take about 8 days to calculate the Jaccard index on all pairs of the 400,000 ads, and take about 640GB of memory to store it. While this is tractable we can find almost all pairs with a significant overlap it in half an hour in-memory using MinHash.

Mathematicians and software developers have a lot in common. They both build structures of ideas, typically working in small groups or alone, but leveraging structures built by others. For software developers the ideas are concrete code implementations, and the building blocks are subroutines, and are published as "libraries" or "packages". For mathematicians the ideas are abstract, built on definitions and theorems and published in papers, conferences and informal conversations. To grow a substantial body of work in both mathematics or software requires a community to contribute to it.

Dealing with thousands of different items is difficult. When you've got a couple of dozen you can view them together, but as you get into the hundreds, thousands and beyond it becomes necessary to group items to make sense of them. For example if you've got a list of customers you might group them by state, or by annual spend. But sometimes it would be useful to split them into a few groups using some heuristic criteria; clustering is a powerful technique to do this.

A decision tree is a series of conditional rules leading to an outcome. When stated as a chain of if-then-else rules it can be really hard to understand what is going on. If the number of dimensions and cutpoints is relatively small it can be useful to visualise on a grid to understand the tree. Decision trees are often represented as a hierarchy of splits. Here's an example of a classification tree on Titanic survivors.

Analysts tend to be natural problem solvers, good at reasoning and adept with numbers. But to know how to frame the problem and what to look for they need to understand the context. To solve the problems they have to collect the right data and perform any necessary calculations. To have impact they need to be able to understand what's valuable, communicate their insights and influence decisions. These make up the four competencies of an effective analyst.

When you've got a timeseries that doesn't have a timezone attched to it the natural question is "what timezone is this data from?" Sometimes it's UTC, sometimes it's the timezone of the server, otherwise it could be the timezone of one of the locations it's about (and it may or may not change with daylight savings). When it's people's web activity there's a simple heuristic to check this: the activity will be minimum between 3am and 5am.

A very useful open dataset the Australian Government provides is the Geocoded National Address File (G-NAF). This is a database mapping addresses to locations. This is really useful for applications that want to provide information or services based on someone's location. For instance you could build a custom store finder, get aggregate details of your customers, or locate business entities with an address, for example ATMs. There's another open and editable dataset of geographic entities, Open Street Map (and it has a pretty good open source Android app OsmAnd).

Sometimes I get out pipe tables in Emacs that I want to convert into a CSVto put somewhere else. This is really easy with regular expressions. I often get data output from an SQL query like this text | num | value --------------+------+------------- Some text | 0.3 | 0.2 Rah rah | 7 | 0.00123(2 rows) Running sed 's/\(^ *\| *|\|(.*\) */,/g' gives: ,text,num,value --------------+------+------------- ,Some text,0.3,0.2 ,Rah rah,7,0.00123, I can delete the divider and then use as a CSV.

Generally when combining datasets you want to join them on some key. But sometimes you really want a range lookup like Excel's VLOOKUP. A common example is binning values; you want to group values into custom ranges. While you could do this with a giant CASE statement, it's much more flexible to specify in a separate table (for regular intervals you can do it with some integer division gymnastics). It is possible to implement VLOOKUP in SQL by using window functions to select the right rows.

A geometric way to represent combining two independent discrete random variables is as a probability square. On each side of the square we have the distributions of the random variables, where the length of each segment is proportional to the probability. In the centre we have the function evaluated on the two edges and the probability is proportional to the area of the rectangle. For example suppose we had a random process that generated 1, 2 or 3 with equal probability (for example half the value of a die, rounded up).

Behavioural data can illuminate the structure of the underlying actors. For example looking at which products customers buy can help understand how both the products and customers interact. The same idea can apply to people who attend events, watch the same movie, or have authored a scientific paper together. There are a few ways to represent these kinds of interactions which gives a large toolbox of ways to approach the problem.

Putting numeric data into bins is a useful technique for summarising, especially for continuous data. This is what underlies histograms which is a bar chart of frequency counts in each bin. There are two main ways of doing this in Excel with groups and with vlookup (you can also do this in SQL). If you want equal length bins in a Pivot Table the easiest way is with groups. Right click on the column you want to bin and select Group

I had to debug some Powershell, without knowing anything about it. I found Write-Warning was the right tool for printline debugging. This was enough to resolve my issue. I first tried Write-Output but apparently it doesn't work inside a function which I found misleading for a while (at first I thought that it wasn't getting to the function). Write-Warning worked straight away and I could see in bright yellow what was going on.

Any analysis where the results won't change a decision is worthless. Before even thinking of getting any data it's worth being clear on how it impacts the decision. There's lots of reasons people want an analysis. Sometimes it's to confirm what they already believe (and they'll discount anything that tells them otherwise). Sometimes it's to prove to others something they believe; possibly to inform a decision someone else is making. But it's most valuable when it effects a decision they can make with an outcome they care about.

I've looked at finding near duplicate job ads using the Jaccard index on n-grams. I wanted to see whether using the TF-IDF to weight the ads would result in a clearer separation. It works, but the results aren't much better, and there are some complications in using it in practice. When trying to find similar ads with the Jaccard index we looked at the proportion of n-grams they have in common relative to all the n-grams between them.

Finding near-duplicate texts is a hard problem, but the Jaccard index for n-grams is an effective measure that's efficient on small sets. I've tried it on the Adzuna Job Salary Predictions Kaggle Competition with good success. This works pretty well at finding near-duplicates and even ads from the same company; although by itself it can't detect duplicates. I've looked before at using the edit distance which looks for the minimum number of changes to transform one text to another, but it's slow to calculate.

Edit distance, also known as Levenshtein Distance is a useful way of the similarity of two sequences. It counts what is the minimum number of substitutions, insertions and deletions you need to make to transform one sequence to another. I had a look at using this for trying to compare duplicate ads with reasonable results, but it's a little slow to run on many ads. I've previously looked at finding ads with exactly the same text in the Adzuna Job Salary Predictions Kaggle Competition, but there are a lot of ads that are slight variations.

Getting Emacs to work nicely on a Windows system can be a challenge. You can install it natively (although getting all the dependencies is a challenge), but many packages require libraries or utilities that are hard to install or don't exist on Windows. The best solution I have found is using Emacs under the Windows Subsystem for Linux (WSL) with Xming. However if you run Emacs 26 or greater after starting Xming with XLaunch you're faced with a blank screen and can't see any writing on Emacs

Two similar documents are likely to have many similar phrases relative to the number of words in the document. In particular if you're concerned with plagiarism and copyright, getting the same data through multiple sources, or finding versions of the same document this approach could be useful. In particular MinHash can quickly find pairs of items with a high Jaccard index, which we can run on sequences of w tokens. A hard question is what's the right number for w?

Finding exact duplicates texts is quite straightforward and fast in Python. This can be useful for removing duplicate entries in a dataset. I tried this on the Adzuna Job Salary Predictions Kaggle Competition job ad texts and found it worked well. Naively finding exact duplicates by comparing every pair would be O(N^2), but if we sort the input, which is O(N log(N)), then duplicate items are adjacent. This scales really well to big datasets, and then the duplicate entries can be handled efficiently with itertools groupby to do something like uniq.

When comparing two texts it's useful to have a side-by-side comparison highlighting the differences. This is straightforward using HTML in Jupyter Notebooks with Python, and the inbuilt DiffLib. I used this to display job ads duplicated between different sites. For a long document it's important to align the sentences (otherwise it's hard to compare the differences), and highlight the individual differences at a word level. Overall the problems are breaking up a text into sentences and words, aligning the sentences, finding word level differences and displaying them side-by-side.

I created a simple custom recipe to show diffs between two texts in Prodigy. I intend to use this to annotate near-duplicates. The process was pretty easy, but I got tripped up a little. I've been extracting job titles and skills from the job ads in the Adzuna Job Salary Predictions Kaggle Competition. One thing I noticed is there are a lot of job ads that are almost exactly the se; sometimes between the train and test set which is a data leak.

For anyone who wants to learn Statistics and has a maths or physics I highly recommend Larry Wasserman's All of Statistics . It covers a wide range of statistics with enough mathematical detail to really understand what's going on, but not so much that the machinery is overwhelming. What I learned reading it really helped me understand statistics well enough to design bespoke statistical experiments and effectively use and implement machine learning models.

Sequences of words are useful for characterising text and for understanding text. If two texts have many similar sequences of 6 or 7 words it's very likely they have a similar origin. When splitting apart text it can be useful to keep common phrases like "New York" together rather than treating them as the separate words "New" and "York". To do this we need a way of extracting and counting sequences of words.

When trying to install packages with apt on a new Ubuntu AWS EC2 instance I had issues where the signature would fail to verify. The reason was the system clock was far in the past and so it looked like the signature was signed in the future. I created a workaround to wait for the system clock to synchronise that solved the problem and could be useful when starting a new machine with time sensitive issues.

I've been trying to use Named Entity Recogniser (NER) to extract job titles from the titles of job ads to better understand a collection of job ads. While NER is great, it's not the right tool for this job, and I'm going to switch to a counting based approach. NER models try to extract things like the names of people, places or products. SpaCy's NER model which I used is optimised to these cases (looking at things like capitalisation of words).

Over the past decade deep neural networks have revolutionised dealing with unstructured data. Problems like identifying what objects are in a video through generating realistic text to translating speech between languages that were intractable are now used in real-time production systems. You might think that today all problems on text, audio an images should be solved by training end-to-end neural networks. However rules and pipelines are still extremely valuable in building systems, and can leverage the information extracted from the black-box neural networks.

Active learning reduces the number of annotations you have to make by selecting for annotation the items that will have the biggest impact on model retraining. Active learning for NER is built into Prodigy, but I failed to use to it to improve my job title recogniser. Having built a reasonable NER model for recognising job titles I wanted to see if I could easily improve it with Protidy's active learning.

In a couple of hours I trained a reasonable job title Named Entity Recogniser for job ad titles using Prodigy, with over 70% accuracy. While 70% doesn't sound great it's a bit ambiguous what a job title is, and getting exactly the bounds of the job title can be a hard problem. It's definitely good enough to be useful, and could be improved. After thinking through an annotation scheme for job titles I wanted to try annotating and training a model.

When doing Named Entity Recognition it's important to think about how to set up the problem. There's a balance between what you're trying to achieve and what the algorithm can do easily. Coming up with an annotation scheme is hard, because as soon as you start annotating you notice lots of edge cases. This post will go through an example with extracting job titles from job ads. In our previous post we looked at what was in a job ad title and a way of extracting some common job titles from the ads.

The job title should succinctly summarise what the role is about, so it should tell you a lot about the role. However in practice job titles can range from very broad to very narrow, be obscure or acronym-laden and even hard to nail down. They're even hard to extract from a job ad's title - which is what I'll focus on in this series. In a previous series of posts I developed a method that could extract skills written a very particular way.

When your harddrive is filling up the du utility is a great way of seeing what's taking up all the space. It can recursively walk through directories to a maximum depth, and print it in human readable sizes. I'll normally start by running df to see what space is used and available. It's worth looking at the Mounted On column if you don't administer the machine because sometimes there are special partitions for large files.

When there's something unexpected happening in your Python code the first thing you want to do is to get more information about what's going wrong. While you can use print statements or logging it may take a lot of iterations of rerunning and editing your statements to capture the right information. You could use a REPL but sometimes it's challenging to capture all the state at the point of execution. The most powerful tool for this kind of problem is a debugger, and it's really easy to get started with Python's pdb.

Moving averages can help smooth out the noise to reveal the underlying signal in a dataset. As they lag behind the actual signal they tradeoff timeliness for increased precision in the underlying signal. You could use them for reporting metrics or for alerting in cases where it's more important to be sure there is a change than it is to catch any change early. It's typically better to have a 7 day moving average than weekly reporting for important metrics because you'll see changes earlier.

Presto and the AWS managed alternative Amazon Athena have some powerful aggregation functions that can make writing SQL much easier. A common problem is getting the most recent status of a transaction log. The max_by function (and its partner min_by) makes this a breeze. Suppose you have a table tracking user login activity over time like this: country user_id time status AU 1 2020-01-01 08:00 logged-in CN 2 2020-01-01 09:00 logged-in AU 1 2020-01-01 12:00 logged-out AU 1 2020-01-01 13:00 logged-in CN 2 2020-01-01 14:00 logged-out You need to find out which users are currently logged in and out, which requires you to find their most recent status.

I find it really handy to have my calendar and contacts from my email client on my mobile phone. DAVx5 is a fantastic free (GPLv3) app to do this on Android. This lets me organise my life accross devices and helps me know when friends and family's birthdays are. DAVx5 is simple to set up and has worked almost flawlessly for me for over 4 years. It supports two way synchronisation to CalDAV and CardDAV servers that many email providers support.

I do a lot of work in Emacs and at the command line, and I get quite a few emails so it would be great if I could handle my emails there too. Email in Emacs can be surprisingly featureful and handles HTML markup, images and can even send org markup with images and equations all from the comfort of an Emacs buffer. However it can be a whole heap of work, and as you get deeper into the features your mail client provides the amount of custom integration required grows very rapidly.

There are many great tools for filtering, transforming and aggregating data like SQL, R dplyr and Python Pandas (not to mention Excel). But sometimes when I'm working on a remote server I want to quickly extract some information from a file without switching to one of these environments. The standard UNIX tools like uniq, sort, sed and awk can do blazing fast transformations on text files that don't fit in memory and are easy to chain together.

Property based testing lets you specify rules that a function being tested will satisfy over a wide range of inputs. This specifies how to thoroughly test a function without coming up with a detailed set of test cases. For example instead of writing a specific test case like sort([1, 3, 2]) == [1, 2, 3], you could state that the input and output of sort should contain exactly the same elements for any valid input.

Dumb-jump is a fantastic emacs package for code navigation. It jumps to the definition of a function/class/variable by searching for regular expressions that look like a definition using ag, ripgrep or git-grep/grep. Because it is so simple it works in over 40 languages (including oddities like SQL, LaTeX and Bash) and is easy to extend. While it is slower and less accurate than ctags, for medium sized projects it's fast enough and requiring no setup makes it much more useful in practice.

I find having Emacs as a unified programming environment really useful. When writing an SQL pipeline I can iteratively develop my SQL in emacs, running it against the database. For a quick and dirty analysis I can copy the output into the .sql file and comment it out. Then I can copy the SQL into a programming language, parameterise it, and test it without touching the mouse. So when I started using Presto and AWS's managed alternative Athena, I needed to integrate it into emacs.

Creating state of the art deep learning algorithms often requires changing the details of the training process. Whether it's scheduling hyperparameters, running on multiple GPUs or plotting the metrics it requires changing something in the training loop. However constantly modifying the core training loop every time you want to add a feature, and adding a switch to enable it, quickly becomes unmaintainable. The solution fast.ai developed is to add points where custom code can be called that modifies the state of training, which they call callbacks.

There are many things that are valuable to know in business but are hard to measure. For example the time from when a customer has a need to purchase, the number of related products customers use or the or the actual value your products are delivering. However you don't need a sample size of hundreds to get an estimate; in fact you can get a statistically significant result from measuring just 5 random customers.

One necessary hurdle in doing data analysis or machine learning is loading the data. In many businesses larger datasets live in databases, in an object store (like Amazon S3) or the Hadoop File System. For some use cases you can do the work where the data lives using SQL or Spark, but sometimes it's more convenient to load it into a language like Python (or R) with a wider range of tools.

I'm trying to extract skills from job ads, using job ads in the Adzuna Job Salary Predictions Kaggle Competition. In the previous post I extracted skills written in phrases like "experience in telesales" using spaCy's dependency parse, but it wouldn't extract many types of experience from a job ad. Here we will extend these rules to extract lists of skills (for example extracting "telesales" and "callcentre" from "experience in telesales or receptionist", which will let us analyse which experiences are related.

Extracting Experience in a Field I'm trying to extract skills from job ads, using job ads in the Adzuna Job Salary Predictions Kaggle Competition. In the previous post I extracted skills written in phrases like "subsea cable engineering experience". This worked well, but extracted a lot of qualifiers that aren't skills (like "previous experience in", or "any experience in"). Here we will write rules to extract experience from phrases like "experience in subsea cable engineering", with much better results.

I'm trying to extract skills from job ads, using job ads in the Adzuna Job Salary Predictions Kaggle Competition. Using rules to extract noun phrases ending in experience (e.g. subsea cable engineering experience) we can extract many skills, but there's a lot of false positives (e.g. previous experience) You can see the Jupyter notebook for the full analysis. Extracting Noun Phrases It's common for ads to write something like "have this kind of experience":

I attended the excellent 2019 Leading the Product conference in Melbourne with aronud 500 other Product Managers and Enthusiasts. The conference had a broad range of great talks, a stimulating networking event where we connected by sharing our favourite books on product management, and overall a energetic atmosphere. I got something out of every talk, but here are the highlights from a data perspective. Find quick ways of testing difficult and uncertain hypotheses John Zeratsky talked about the design sprint for implementing a design solution in a week; from storyboarding an experience, to brainstorming solutions to prototyping and testing.

When predicting outcomes using machine learning it's always useful to have a baseline to compare results against. A simple baseline is the best constant model; that is a model that gives the same prediction for any input. This is a really simple check to perform against any dataset, and can be informative to check across validation splits. There are simple algorithms for finding the best constant model. For categorical predictions just evaluate every possible category to choose as the constant prediction.

Intro When I was 15 I did a week of work experience with my neighbour, who was an agricultural economist running his own one person business. I'm still not really sure what an agricultural economist does, but I went out with him to visit his clients to talk through their business, and saw how he analysed their data in his Excel spreadsheet. It was really closer to an application than a spreadsheet; the interface made it clear where the client was meant to enter their data, it showed some summary output and most of the intermediate calculations were hidden.

Why is a sodium lamp yellow? How can we determine the elemental composition of the sun? How does a Helium-neon laser can work? To some degree all of these questions require knowing the spectra of atoms, which can in theory be calculated by Quantum mechanics. However the calculations of these spectra for arbitrary systems from first principles is prohibitively difficult and computationally intensive (which is why techniques such as Density Functional Theory are used).

There are two main constructions of a projective space.

This post is based mainly on a chapter in A Radical Approach to Lebesgue’s Theory of Integration by David Bressoud in which he explores the history of the Lebesgue integral. The story I will tell is closer to folklore than a historical account, but nevertheless enlightening.

In this article I will show that the cyclic group of order n, that is the set \(\{0,1,2,\ldots,n-1\}\) under addition modulo n motivates the discrete Fourier transform on a particular finite dimensional complex inner product space, and gives many of its properties. In a subsequent article I will extend this to the general Fourier transform and its relation to the group of integers and real numbers under addition.

There is a deep relationship between the technique of separation of variables for solving partial differential equations and the symmetries of the underlying differential equations, as well as the special functions that often arise in this procedure.

In this article I will apply the ideas from part 1 to the theory of rotations in three dimensions. (The theory of rotations in an arbitrary number of dimensions is similar, but for reasons of familiarity and simplicity I will stick to 3 dimensions).

There is a deep relationship between being able to solve a differential equation and its symmetries. Much of the theory of second order linear differential equations is really the theory of infinite dimensional linear algebra. In particular Sturm-Liouville theory is the diagonalization of an infinite dimensional Hermitian operator. However there are deeper relationships, as Miller points out in “Lie theory and special functions”; the relationships between special functions such as Rodrigues’ formulae are related to the Lie algebra and symmetries of the system. Even better in some cases the solutions can be found almost entirely algebraically. Some examples from physics come from the Simple Harmonic Oscillator, the theory of Angular Momentum and the Kepler Problem (using the Laplace Runge Lenz vector). The rest of this article will be devoted to exploring a special case of these relations the Quantum Simple Harmonic Oscillator.

In mathematics and physics it is common to talk about \(\mathbb{R}^n\) when really we mean something else that can be represented by \(\mathbb{R}^n\).

Consider mechanics or geometry, these are often represented as theories in \(\mathbb{R}^n\) , but really they don’t occur in a vector space at all! Look around you, a three-dimensional description of space probably seems reasonable, but where’s the origin? [Perhaps the centre of your eyes could be an origin, but someone else would disagree with you]. Classical mechanics, special relativity and geometry are much better described as an affine space – which is a vector space without an origin.

Often in mathematics there is the idea of taking the closure of some elements under a particular operation. For instance if you have several vectors you may take the span of them, if you have a field and want to add the zeros of a particular polynomial you want to consider the corresponding extension field, given a set you often want to construct “the finest topology such that…”.

\[\newcommand\nth{n^{\mathrm{th}}}\]

It is well known in mathematics that it is possible to find the roots of a general quadratic, cubic or quartic in terms of radicals (linear combinations and products of \(\nth\) roots). Another way of saying this is that the equation \(a x^4+b x^3+c x^2 + d x + e = 0\) can be solved for any complex constants \(a\),\(b\),\(c\),\(d\), and \(e\) if one can solve the equation \(x^n-t=0\) for \(n \in \{2,3\}\) (\(1\) being trivial) (\(t\) may be an algebraic combination of solutions of \(x^n-s\) for a variety of \(s\) which are algebraic combinations of \(a\),\(b\),\(c\),\(d\) and \(e\)). This is not true for the quintic.

I wanted to do the contour integral of \(\frac{1}{z-a}\) around the unit circle on a computer for kicks. So I parameterised it with \(e^{it}\) and entered it into Maxima and Matlab

\[\int_0^{2\pi} \frac{i e^{it}}{e^{it}-a} dt\]

for various values of \(a\) . By Cauchy’s integral formula we know it should be \(2 \pi i\) if \(|a|<1\) and 0 if \(|a|>1\) . Interestingly both programs gave the right answer for \(a=0\) (I suppose the calculation is easy there) but gave totally wrong answers for \(a \neq 0\) (Matlab gave \(0\) for \(|a|<1\) and \(-2 \pi i\) if \(|a|>1\) and Maxima gave \(0\) everywhere).

I have no idea why this happens! In Matlab if you expand the complex exponential it gives the right answer. In Maxima it then has trouble computing the integral – but if you perform operations to make the denominator real and perform some simplifications it gives the right answer.

Now people sometimes ask Why learn arithmetic when we have calculators? I think this shows exactly why: you need to know when your calculator is giving you garbage.

Lie Groups are mathematically “very nice” structures – they are analytic manifolds (real or complex) with a group structure such that multiplication and inversion are continuous. They are deeply related to infinitesimal symmetries; a group acting on a space can generally be thought of as a group of symmetries (or automorphisms) of some structure [e.g. rotations (\(SO(n)\)) preserve lengths, Möbius transformations map circles and lines in the complex plane into circles and lines], and the analyticity gives a “local” feel to it – if we know the symmetry locally we can extend it analytically (by the exponential map).

Hilbert’s Fifth Problem essentially asked how restrictive just looking at analytic actions is – what if we looked at continuous actions, how many more groups would we get?

Theorem [Gleason, Montgomery, Zippin] For a locally compact group \(G\) the following are equivalent:

1. \(G\) is locally Euclidean.
2. \(G\) has no small subgroups; i.e. there exists a neighbourhood of the identity that contains no non-trivial subgroups of \(G\).
3. \(G\) is a Lie Group.