Finding Types of Experience from Adzuna Job Ads

We’re going to use linguistic features to extract the types of experience commonly required for jobs from job ads. I’m not exactly sure what I mean by “types of experience”; we’re going to let the data decide that!

We’ll end up with a list of skills, and some relationships between skills that occur together

Import libraries and data

import re
import pandas as pd
import spacy
from spacy.util import filter_spans
from spacy.tokens import Span
from spacy.matcher import Matcher

spacy.__version__

'2.2.3'

from spacy import displacy
from IPython.display import HTML, display

Get the data from Adzunda Job Salary Prediction Kaggle Competition, put it in the data subfolder and unzip all the files.

You can do this manually, or use the Kaggle API (once you’ve installed the API, downloaded your kaggle.json file and agreed to the competition rules)

# for split, ext in [('Test', 'zip'), ('Train', 'zip'), ('Valid', 'csv')]:
#     !kaggle competitions download -c job-salary-prediction --path data/ -f {split}_rev1.{ext}
    
# !find data/ -name '*.zip' -execdir unzip '{}' ';'
# !find data/ -name '*.zip' -exec rm '{}' ';'

# !ls data/

Read in all the data to a single dataframe

%%time
dfs = []
for split in ['Train', 'Valid', 'Test']:
    dfs.append(pd.read_csv(f'data/{split}_rev1.csv').assign(split=split))
df = pd.concat(dfs, sort=False, ignore_index=True)
del dfs

CPU times: user 6.55 s, sys: 2.8 s, total: 9.34 s
Wall time: 16.1 s

Train/Valid/Test is in the ratio 6:1:3, with about 40k ads in total

df.split.value_counts()

Train    244768
Test     122463
Valid     40663
Name: split, dtype: int64

We’re mainly interested in the ad content where the skills will be; that’s the FullDescription

df

	Id	Title	FullDescription	LocationRaw	LocationNormalized	ContractType	ContractTime	Company	Category	SalaryRaw	SalaryNormalized	SourceName	split
0	12612628	Engineering Systems Analyst	Engineering Systems Analyst Dorking Surrey Sal...	Dorking, Surrey, Surrey	Dorking	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 30000/annum 20-30K	25000.0	cv-library.co.uk	Train
1	12612830	Stress Engineer Glasgow	Stress Engineer Glasgow Salary ** to ** We...	Glasgow, Scotland, Scotland	Glasgow	NaN	permanent	Gregory Martin International	Engineering Jobs	25000 - 35000/annum 25-35K	30000.0	cv-library.co.uk	Train
2	12612844	Modelling and simulation analyst	Mathematical Modeller / Simulation Analyst / O...	Hampshire, South East, South East	Hampshire	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 40000/annum 20-40K	30000.0	cv-library.co.uk	Train
3	12613049	Engineering Systems Analyst / Mathematical Mod...	Engineering Systems Analyst / Mathematical Mod...	Surrey, South East, South East	Surrey	NaN	permanent	Gregory Martin International	Engineering Jobs	25000 - 30000/annum 25K-30K negotiable	27500.0	cv-library.co.uk	Train
4	12613647	Pioneer, Miser Engineering Systems Analyst	Pioneer, Miser Engineering Systems Analyst Do...	Surrey, South East, South East	Surrey	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 30000/annum 20-30K	25000.0	cv-library.co.uk	Train
...	...	...	...	...	...	...	...	...	...	...	...	...	...
407889	72703426	Foreign Exchange Consultant Worcestershire	Do you have foreign exchange cashier experienc...	Worcestershire	Worcestershire	full_time	permanent	Travel Trade Recruitment	Travel Jobs	NaN	NaN	jobs.travelweekly.co.uk	Test
407890	72703453	Senior Business Travel Consultant	Senior Business Travel Consultant Birmingham ...	Birmingham	Birmingham	full_time	permanent	AA Appointments	Travel Jobs	NaN	NaN	jobs.travelweekly.co.uk	Test
407891	72705210	TEACHER OF MATHS	Position: Qualified Teacher Subject/Specialism...	Swindon	Swindon	NaN	contract	NaN	Teaching Jobs	NaN	NaN	hays.co.uk	Test
407892	72705214	Welsh Speaking Teaching Assistant Job	Hays Education currently have a job for a Wels...	Cardiff	Cardiff	NaN	contract	NaN	Teaching Jobs	NaN	NaN	hays.co.uk	Test
407893	72705218	KS2 Teacher	Are you a School KS2 Teacher looking for temp ...	Camberley	Camberley	NaN	contract	NaN	Teaching Jobs	NaN	NaN	hays.co.uk	Test

407894 rows × 13 columns

Extract the ads into a list

ads = list(df.FullDescription)

Initialise Spacy model

nlp = spacy.load('en_core_web_lg')

Extracting from job ads

Let’s look at sentences in job ads containing the word ‘experience’.

Experience is a common word, but used in a few different ways:

has experience with a tool/using a skill/in a system
providing an experience to customers
this job will give you experience

We’re interested in the first kind which occurs in a few different ways:

{type of} experience …
experience in {field}

Let’s look at extracting them

def highlight_terms(terms, texts):
    for doc in nlp.pipe(texts):
        for sentence in set([tok.sent for tok in doc if tok.lower_ in terms]):
            text = sentence.text.strip()
            markup = re.sub(fr'(?i)\b({"|".join(terms)})\b', r'<strong>\1</strong>', text)
            display(HTML(markup))

Note that you can already see some problems with the way the text was cleansed; it looks like list structure is gone and hyphens have been removed (35 years experience is probably 3-5 years experience).

highlight_terms(['experience'], ads[:10])

Aerostructures experience You will also be expected to demonstrate the following qualities: A strong desire to progress quickly to a position of leadership Professional approach Strong communication skills, written and verbal Commercial awareness Team working, being comfortable working in international teams and self managing

The Requirements You will need to have a good engineering degree that includes structural analysis (such as aeronautical, mechanical, automotive, civil) with some experience in a professional engineering environment relevant to (but not limited to)

You will need to demonstrate experience in at least one or more of the following areas:

The roles are ideally suited to high calibre engineering graduates with any level of appropriate experience, so that we can give you the opportunity to use your technical skills to provide high quality input to our aerospace projects, spanning both aerostructures and aeroengines.

Thorough knowledge of Excel and proven ability to utilise this to create powerful decision support models Experience in Modelling and Simulation Techniques, Experience of techniques such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background

*K AAE pension contribution, private medical and dental The opportunity Our client is an independent consultancy firm which has an opportunity for a Data Analyst with 35 years experience.

In addition to formal qualifications and experience, the successful candidate will require excellent written and verbal communication skills, be energetic, enterprising and have a determination to succeed.

Any experience of Pioneer or Miser software would be an advantage.

This role will be working within a small team working on the modelling of water industry asset deterioration and asset failure consequences, including the uploading of these models onto industryleading optimal asset management software Strong maths, stats and IT skills needed, Any previous experience within the Water industry would be an advantage.

For this role, you must have a minimum of 10 years experience in subsea engineering, pipelines design or construction.

Naturally positive and looking for the opportunity to work for a national player who are committed to your success Recruitment experience or 2 years business to business sales experience required.

This is an exceptional opportunity to join a construction / technical agency that hasn t shrunk in the current market one bit Our client is seeking a nononsense and highly skilled Recruiter with at least a couple of years experience under their belt.

They will need someone who has at least 1015 years of subsea cable engineering experience with significant experience within offshore oil and gas industries.

Helper functions

Let’s take a variety of informative examples to test extractions on

examples = [
    'They will need someone who has at least 1015 years of subsea cable engineering experience',
    'This position is ideally suited to high calibre engineering graduate with significant and appropriate post graduate experience.',
    'Aerospace industry experience would be advantageous covering aerostructures and/or aero engines.',
    'A sufficient and appropriate level of building services and controls experience gained within a client organisation, engineering consultancy or equipment supplier.',
    
    'Experience in Modelling and Simulation Techniques',
    'Any experience of Pioneer or Miser software would be an advantage.',
    'For this role, you must have a minimum of 10 years experience in subsea engineering, pipelines design or construction.',
    'Has experience within the quality department of a related company in a similar role Ideally from a mechanical or manufacturing engineering background.',
    'and have experience of the technical leadership of projects to time, quality and cost objectives.',
    'Experience of protection and control design at Transmission and Distribution voltages.',
    'Candidates with experience in telesales, callcentre, customer service, receptionist or travel are ideal for this role',
    'Experience dealing with business clients (B2B) would be preferable.',
    'Previous experience working as a Chef de Partie in a one AA Rosette hotel is needed for the position.',
    'The post holder must hold as a minimum Level 1 in Trampolining (British Gymnastics) and have experience in working with children, be fun, outgoing and have excellent customer service skills and be able to instruct in line with the British Gymnastics syllabus.',
    'Experience of techniques such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background',

    
]

We could look to extract:

a series of nouns before the word experience (e.g. “subsea cable engineering experience”); or
experience as/in something (e.g “experience as a Chef de Partie”)

we’ll do this using Spacy’s Rule Based Matcher

matcher = Matcher(nlp.vocab)
pattern = [{'POS': 'NOUN', 'OP': '+'}, {'LOWER': 'experience'}]
matcher.add('experience_noun', [pattern])

pattern = [{'LOWER': 'experience'}, {'POS': 'ADP'}, {'POS': {'IN': ('DET', 'NOUN', 'PROPN')}, 'OP': '+'}]
matcher.add('experience_adp', [pattern])

doc = nlp(examples[0])
matcher(doc)

[(12285600890577657150, 13, 15), (12285600890577657150, 12, 15)]

Here we have a little helper function to visualise extractions.

def show_extraction(examples, *extractors):
    seen = set()
    for doc in nlp.pipe(examples):
        doc.ents = filter_spans([Span(doc, start, end, label) for extractor in extractors for label, start, end in extractor(doc)])
        for tok in doc:
            if tok.lower_ == 'experience':
                sentence = tok.sent
                if sentence.text in seen:
                    continue
                seen.update([sentence.text])
                if not sentence.ents:
                    doc.ents = list(doc.ents) + [Span(doc, tok.i, tok.i+1, 'MISSING')]
                displacy.render(sentence, style='ent', options = {'colors': {'MISSING': 'pink',
                                                                            'EXPERIENCE': 'lightgreen'}})

This is on the right track, but doesn’t always pick up the appropriate context.

show_extraction(examples, matcher)

They will need someone who has at least 1015 years of subsea cable engineering experience experience_noun

This position is ideally suited to high calibre engineering graduate with significant and appropriate post graduate experience experience_noun .

Aerospace industry experience experience_noun would be advantageous covering aerostructures and/or aero engines.

A sufficient and appropriate level of building services and controls experience experience_noun gained within a client organisation, engineering consultancy or equipment supplier.

Experience in Modelling experience_adp and Simulation Techniques

Any experience of Pioneer experience_adp or Miser software would be an advantage.

For this role, you must have a minimum of 10 years experience experience_noun in subsea engineering, pipelines design or construction.

Has experience within the quality department experience_adp of a related company in a similar role Ideally from a mechanical or manufacturing engineering background.

and have experience of the experience_adp technical leadership of projects to time, quality and cost objectives.

Experience of protection experience_adp and control design at Transmission and Distribution voltages.

Candidates with experience in telesales experience_adp , callcentre, customer service, receptionist or travel are ideal for this role

Experience MISSING dealing with business clients (B2B) would be preferable.

Previous experience MISSING working as a Chef de Partie in a one AA Rosette hotel is needed for the position.

The post holder must hold as a minimum Level 1 in Trampolining (British Gymnastics) and have experience MISSING in working with children, be fun, outgoing and have excellent customer service skills and be able to instruct in line with the British Gymnastics syllabus.

Experience of techniques experience_adp such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background

We can then extract them from a document.

Note the use of filter_spans; this ensures if we have overlapping spans we only take the largest one.

def get_extractions(examples, *extractors):
    # Could use context instead of enumerate
    for idx, doc in enumerate(nlp.pipe(examples, batch_size=100, disable=['ner'])):
        for ent in filter_spans([Span(doc, start, end, label) for extractor in extractors for label, start, end in extractor(doc)]):
            sent = ent.root.sent
            yield ent.text, idx, ent.start, ent.end, ent.label_, sent.start, sent.end

list(get_extractions(ads[:3], matcher))

[('experience in a', 1, 150, 153, 'experience_adp', 122, 164),
 ('years experience', 2, 45, 47, 'experience_noun', 16, 48),
 ('decision support models Experience', 2, 92, 96, 'experience_noun', 79, 118),
 ('Experience of techniques', 2, 102, 105, 'experience_adp', 79, 118)]

Put it in a dataframe and join with the job metadata

def extract_df(*extractors, n_max=None, **kwargs):
    if n_max is None:
        n_max = len(df)
    ent_df = pd.DataFrame(list(get_extractions(df[:n_max].FullDescription, *extractors)),
                          columns=['text', 'docidx', 'start', 'end', 'label', 'sent_start', 'sent_end'])
    return ent_df.merge(df, how='left', left_on='docidx', right_index=True)

%time ent_df = extract_df(matcher, n_max=1000)
ent_df.head()

CPU times: user 16.2 s, sys: 11.7 s, total: 27.9 s
Wall time: 1min 18s

	text	docidx	start	end	label	sent_start	sent_end	Id	Title	FullDescription	LocationRaw	LocationNormalized	ContractType	ContractTime	Company	Category	SalaryRaw	SalaryNormalized	SourceName	split
0	experience in a	1	150	153	experience_adp	122	164	12612830	Stress Engineer Glasgow	Stress Engineer Glasgow Salary ** to ** We...	Glasgow, Scotland, Scotland	Glasgow	NaN	permanent	Gregory Martin International	Engineering Jobs	25000 - 35000/annum 25-35K	30000.0	cv-library.co.uk	Train
1	years experience	2	45	47	experience_noun	16	48	12612844	Modelling and simulation analyst	Mathematical Modeller / Simulation Analyst / O...	Hampshire, South East, South East	Hampshire	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 40000/annum 20-40K	30000.0	cv-library.co.uk	Train
2	decision support models Experience	2	92	96	experience_noun	79	118	12612844	Modelling and simulation analyst	Mathematical Modeller / Simulation Analyst / O...	Hampshire, South East, South East	Hampshire	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 40000/annum 20-40K	30000.0	cv-library.co.uk	Train
3	Experience of techniques	2	102	105	experience_adp	79	118	12612844	Modelling and simulation analyst	Mathematical Modeller / Simulation Analyst / O...	Hampshire, South East, South East	Hampshire	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 40000/annum 20-40K	30000.0	cv-library.co.uk	Train
4	experience within the Water industry	5	117	122	experience_adp	71	127	13179816	Engineering Systems Analyst Water Industry	Engineering Systems Analyst Water Industry Loc...	Dorking, Surrey, Surrey, Surrey	Dorking	NaN	permanent	Gregory Martin International	Engineering Jobs	20000 - 30000/annum 20K to 30K	25000.0	cv-library.co.uk	Train

Aggregate the counts of different texts.

It’s more significant if it happens accross multiple Advertisers/Sources.

def aggregate_df(df, col=['text']):
    return (df
            .groupby(col)
            .agg(n_company=('Company', 'nunique'),
                 n_ad=('Id', 'nunique'),
                 n_source=('SourceName', 'nunique'),
                 n=('Id', 'count'))
            .reset_index()
            .sort_values(['n_company', 'n_ad', 'n'], ascending=False)
        )

Unfortunately what is caught with these simple rules has mixed results

aggregate_df(ent_df).head(10)

	text	n_company	n_ad	n_source	n
119	experience in a	4	52	5	52
286	years experience	3	22	3	22
233	management experience	2	17	3	18
69	banqueting experience	2	2	1	2
87	design experience	2	2	1	2
88	development experience	2	2	1	2
196	experience within a	1	7	2	7
260	rosette experience	1	5	2	5
176	experience of the	1	4	2	4
142	experience in software development	1	3	1	3

Let’s add some tooling to look at specific cases

def showent(docidx, start, end, label, sent_start, sent_end, **kwargs):
    # We don't need to parse it, so just make_doc
    doc = nlp.make_doc(ads[docidx])
    doc.ents = [Span(doc, start, end, label)]
    sent = doc[sent_start:sent_end]
    displacy.render(sent, style='ent')
    
def showent_df(df):
    for idx, row in df.iterrows():
        showent(**row)

We can see that we’ve actually missed the subject entirely!

We could be a bit more clever and use some structure from the grammar to extract what we need.

showent_df(ent_df.query('text == "experience in a"').head())

The Requirements You will need to have a good engineering degree that includes structural analysis (such as aeronautical, mechanical, automotive, civil) with some experience in a experience_adp professional engineering environment relevant to (but not limited to)

All Chef de Parties applying for this role must have a strong background with highlights previous AA Rosette experience in a experience_adp high volume operation.

CE hands on experience in a experience_adp nuclear plant environment.

Do you want to have experience in a experience_adp

The ideal Junior Sous will need to have at least two years experience in a experience_adp similar operation you will need to have high standards of presentation and passion for high quality fresh gastro pub food.

Extracting types of experience

Let’s extract some examples of {type of} experience

Here’s a rough rule to extract the phrase to the left of the word ‘experience’ using SpaCy’s noun_chunks, which is based on the syntactic structure (see spacy.lang.en.syntax_iterators)

def extract_noun_phrase_experience(doc):
    for np in doc.noun_chunks:
        if np[-1].lower_ == 'experience':
            if len(np) > 1:
                yield 'EXPERIENCE', np[0].i, np[-1].i

Notice how our rule picks out the right amount of context like “subsea cable engineering”.

However we’re also picking up quantifiers like “Any” and “10 years”

show_extraction(examples, extract_noun_phrase_experience)

They will need someone who has at least 1015 years of subsea cable engineering EXPERIENCE experience

This position is ideally suited to high calibre engineering graduate with significant and appropriate post graduate EXPERIENCE experience.

Aerospace industry EXPERIENCE experience would be advantageous covering aerostructures and/or aero engines.

A sufficient and appropriate level of building services and controls EXPERIENCE experience gained within a client organisation, engineering consultancy or equipment supplier.

Experience MISSING in Modelling and Simulation Techniques

Any EXPERIENCE experience of Pioneer or Miser software would be an advantage.

For this role, you must have a minimum of 10 years EXPERIENCE experience in subsea engineering, pipelines design or construction.

Has experience MISSING within the quality department of a related company in a similar role Ideally from a mechanical or manufacturing engineering background.

and have experience MISSING of the technical leadership of projects to time, quality and cost objectives.

Experience MISSING of protection and control design at Transmission and Distribution voltages.

Candidates with experience MISSING in telesales, callcentre, customer service, receptionist or travel are ideal for this role

Experience MISSING dealing with business clients (B2B) would be preferable.

Previous EXPERIENCE experience working as a Chef de Partie in a one AA Rosette hotel is needed for the position.

Experience MISSING of techniques such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background

Let’s look at how this does across a larger sample of job ads:

There are sentence/word boundary errors that cause the rule to break (e.g. powerful decision support models Experience)
We pick up quantifiers (previous, some, appropriate), as well as time quantifiers (3-5 years, 10 years)

show_extraction(ads[:10], extract_noun_phrase_experience)

The roles are ideally suited to high calibre engineering graduates with any level of appropriate EXPERIENCE experience, so that we can give you the opportunity to use your technical skills to provide high quality input to our aerospace projects, spanning both aerostructures and aeroengines.

The Requirements You will need to have a good engineering degree that includes structural analysis (such as aeronautical, mechanical, automotive, civil) with some EXPERIENCE experience in a professional engineering environment relevant to (but not limited to)

You will need to demonstrate experience MISSING in at least one or more of the following areas:

Aerostructures EXPERIENCE experience You will also be expected to demonstrate the following qualities: A strong desire to progress quickly to a position of leadership Professional approach Strong communication skills, written and verbal Commercial awareness Team working, being comfortable working in international teams and self managing

*K AAE pension contribution, private medical and dental The opportunity Our client is an independent consultancy firm which has an opportunity for a Data Analyst with 35 years EXPERIENCE experience.

Thorough knowledge of Excel and proven ability to utilise this to create powerful decision support models EXPERIENCE Experience in Modelling and Simulation Techniques, Experience of techniques such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background

In addition to formal qualifications and experience MISSING , the successful candidate will require excellent written and verbal communication skills, be energetic, enterprising and have a determination to succeed.

This role will be working within a small team working on the modelling of water industry asset deterioration and asset failure consequences, including the uploading of these models onto industryleading optimal asset management software Strong maths, stats and IT skills needed, Any previous EXPERIENCE experience within the Water industry would be an advantage.

Any EXPERIENCE experience of Pioneer or Miser software would be an advantage.

For this role, you must have a minimum of 10 years EXPERIENCE experience in subsea engineering, pipelines design or construction.

Naturally positive and looking for the opportunity to work for a national player who are committed to your success Recruitment experience or 2 years business to business sales EXPERIENCE experience required.

They will need someone who has at least 1015 years of subsea cable engineering EXPERIENCE experience with significant EXPERIENCE experience within offshore oil and gas industries.

%time ent_df = extract_df(extract_noun_phrase_experience, n_max=50000)

CPU times: user 14min 4s, sys: 6min 43s, total: 20min 48s
Wall time: 25min 59s

Again we are frequently picking up quantifiers

aggregate_df(ent_df).head(50)

	text	n_company	n_ad	n_source	n
13558	previous	894	2040	100	2128
5745	Previous	883	1745	100	1829
6321	Proven	260	468	73	515
15315	some	229	355	65	367
17077	your	221	864	70	874
11492	extensive	220	381	68	389
14616	relevant	208	398	62	405
14298	proven	189	301	63	305
10820	demonstrable	142	217	54	222
16178	the	138	303	56	313
14971	significant	133	207	53	209
15642	strong	125	237	53	240
1810	Any	124	226	40	234
3525	Extensive	123	203	48	225
14849	sales	117	201	35	204
932	2 years	117	195	49	198
11195	equivalent	113	179	41	190
16576	the relevant	105	261	43	263
2946	Demonstrable	104	169	53	186
7462	Strong	102	165	46	169
16438	the following	99	164	47	165
10460	commercial	96	167	33	171
14130	prior	95	163	42	167
15202	solid	93	156	46	161
7320	Some	91	133	37	137
14449	recent	88	213	47	220
11895	good	88	127	43	128
1089	3 years	86	144	44	152
13472	practical	77	99	36	107
7142	Significant	75	136	45	147
17219	•	73	148	26	204
6215	Prior	72	110	37	114
1270	5 years	71	101	38	109
5192	No	70	157	24	166
9746	at least 2 years	68	138	36	138
9542	any	65	111	32	112
3329	Essential	61	110	34	111
12609	management	61	104	36	106
16736	this	59	116	41	118
6655	Relevant	58	92	43	95
2994	Desirable	57	102	35	102
16021	supervisory	56	105	31	106
107	*	56	97	34	117
264	**	56	88	37	96
16666	their	56	86	43	86
7007	Sales	54	98	23	98
7265	Solid	54	77	32	82
6726	Required	53	99	29	99
12320	industry	52	82	29	83
15938	substantial	52	68	32	71

showent_df(ent_df.query("text=='Previous'").head(5))

Previous EXPERIENCE experience is always helpful but not essential, as long as you enjoy looking after people and are able to work evenings and weekends, we would like to hear from you.

Previous EXPERIENCE experience working as a Chef de Partie in a one AA Rosette hotel is needed for the position.

Previous EXPERIENCE experience in major infrastructural projects coupled with the ability to present tenders to the Directors with the technical and commercial skills required for a senior role.

Previous EXPERIENCE experience of working in a business managing large customers and internal stakeholders.

This looks like a bad parse (probably because of the stripped whitespace)

showent_df(ent_df.query("text=='Skills'").head(5))

Skills EXPERIENCE Experience

Skills EXPERIENCE Experience:

The successful candidate will be expected to assist other staff in the Finance Operations team as and when required Skills EXPERIENCE Experience: Good PC skills including MS Excel and Word Strong communication skills including telephone Good organisational skills

Skills EXPERIENCE Experience required: Experience of London market insurance underwriting.

Skills EXPERIENCE Experience

We can blacklist the most common qualifiers

experience_qualifiers = ['previous', 'prior', 'following', 'recent', 'the above', 'past',
                         
                         'proven', 'demonstrable', 'demonstrated', 'relevant', 'significant', 'practical',
                         'essential', 'equivalent', 'desirable', 'required', 'considerable', 'similar',
                         'working', 'specific', 'qualified', 'direct', 'hands on', 'handson', 
                         
                         'strong', 'solid', 'good', 'substantial', 'excellent', 'the right', 'valuable', 'invaluable',
                         
                         'some', 'any', 'none', 'much', 'extensive', 'no', 'more',
                         'your', 'their',
                         'years', 'months',
                         'uk',
                        ]

stopwords = ['a', 'an', '*', '**', '•', 'this', 'the', ':', 'Skills']

experience_qualifier_pattern = rf'\b(?:{"|".join(experience_qualifiers)})\b'

experience_qualifier_pattern

'\\b(?:previous|prior|following|recent|the above|past|proven|demonstrable|demonstrated|relevant|significant|practical|essential|equivalent|desirable|required|considerable|similar|working|specific|qualified|direct|hands on|handson|strong|solid|good|substantial|excellent|the right|valuable|invaluable|some|any|none|much|extensive|no|more|your|their|years|months|uk)\\b'

If we ignore stopwords we start getting some skills out:

sales
commercial
managment
supervisory
customer service
development
supervisory
technical
managment
telesales
financial services
design
project managment
retail
business sales
SQL
marketing
people management
SAP
engineering

aggregate_df(ent_df[(~ent_df.text.str.lower().str.contains(experience_qualifier_pattern)) & # Not a qualifier
                     ~ent_df.text.isin(stopwords)]).head(50)

	text	n_company	n_ad	n_source	n
8436	sales	117	201	35	204
6108	commercial	96	167	33	171
7445	management	61	104	36	106
8733	supervisory	56	105	31	106
4013	Sales	54	98	23	98
7187	industry	52	82	29	83
1311	Commercial	43	68	20	71
9040	the customer	43	65	27	66
6359	customer service	42	58	23	69
9463	work	39	61	29	62
6471	development	28	43	19	44
6346	customer	27	41	22	43
4158	Skills/	26	44	23	45
9353	user	26	39	20	44
2399	Ideally	25	30	17	30
4276	Supervisory	24	34	22	34
8857	telesales	22	38	17	38
67	)	22	32	18	32
8809	technical	22	31	16	31
2859	Management	21	48	26	48
1967	Financial Services	21	31	18	31
6439	design	21	29	19	29
8547	skills	21	29	18	29
8325	registration	20	120	13	120
3616	Project management	20	30	16	30
8386	retail	19	36	21	36
3603	Project Management	19	25	14	25
5954	business sales	19	24	12	24
2649	Knowledge/	18	26	12	30
3931	SQL	18	22	14	24
3667	Qualifications/	17	35	13	35
7496	marketing	17	30	18	30
2461	Industry	17	27	13	27
6973	graduate	17	26	12	29
8166	professional	17	26	14	26
6732	experience	17	23	13	23
1577	Design	17	19	8	19
9138	the necessary	16	54	18	54
7172	indepth	16	28	18	30
4379	Technical	16	20	14	21
1854	Experience	16	18	9	18
7965	people management	15	25	16	25
4723	Work	15	21	15	21
3858	SAP	15	17	11	17
6652	engineering	15	16	11	16
7461	managerial	14	29	18	29
4150	Skills /	14	24	14	24
5663	appropriate	14	24	13	24
5431	agency	14	19	14	20
5531	an advantage	14	18	16	18

Commercial is more of a qualifier

showent_df(ent_df.query("text=='Commercial'").head(5))

Experience Required In order of importance:Substantial ManManagement, departmental, budgetary and Project Management and Commercial EXPERIENCE experience in a large multi country business or operation.

Commercial EXPERIENCE experience of SAS or SQL.

Commercial EXPERIENCE experience of administrating and supporting of Linux and VMWare.

Commercial EXPERIENCE experience in similar role, working with Autodesk drawing package (AutoCAD 2010 or newer)

Commercial EXPERIENCE experience with .Net

Management experience seems correct

showent_df(ent_df.query("text=='Management'").head(5))

You will need to be RMN qualified with Management EXPERIENCE experience within a hospital setting.

To ensure that CQC Standards for Quality & Safety are met at all times You will need to be RMN qualified with Management EXPERIENCE experience within a hospital setting ideally in a charge nurse role

To ensure that CQC Standards for Quality & Safety are met at all times You will need to be RMN qualified with Management EXPERIENCE experience within a hospital setting.

The successful candidate will have proven Management EXPERIENCE experience and hold a 1st level nursing qualification with an active NMC PIN.

You will ideally be RGN or RMN qualified with Management EXPERIENCE experience.

showent_df(ent_df.query("text=='Financial Services'").head(5))

Financial Services EXPERIENCE experience

Financial Services EXPERIENCE Experience

Financial Services EXPERIENCE experience would be beneficial.

Financial Services EXPERIENCE experience

Ability to work to targets Experience of successfully working in a targeted contact centre environment (with Financial Services EXPERIENCE experience) is essential.

showent_df(ent_df.query("text=='SQL'").head(5))

Further refining the physical design to meet system storage requirements Education/Experience Qualifications BS or MS degree or equivalent experience in a technical field Skills: 3 years SQL EXPERIENCE experience 3 years Oracle PL/SQL experience

SQL EXPERIENCE experience using one or more of SQL Server, Oracle or DB

SAS and or SQL EXPERIENCE experience is a key requirement as the role has a heavy analytical bias.

**k Global service provider is looking for a experienced Microsoft Dynamics Solutions Architect with extensive CRM, C, .Net and SQL EXPERIENCE experience.

showent_df(ent_df.query("text=='development'").head(5))

**) Have development EXPERIENCE experience in C and JavaScript

We are offering an exciting opportunity for an experienced applications programmer with a number of years of .Net development EXPERIENCE experience.

Desirable experience would include creating UML diagrams, development EXPERIENCE experience with file systems / database storage, experience with clustered computer environments, or experience with distributed computation platforms.

Permanent The successful candidate will have a minimum of 5 years of development EXPERIENCE experience If you are looking for a new .NET Development challenge and are looking for a market leading Software Company with excellent career progression opportunities and expanding your knowledge and experience, send in your CV now and call Nick Bray on *

It is also desirable that you have had development EXPERIENCE experience in a variety of platforms (Windows, Linux, Android and/or other)

Extracting experience in a field

Another way experience is commonly stated is with an adposition

experience in/with modelling

For example

doc = nlp('Experience of protection and control design at Transmission and Distribution voltages.')
displacy.render([doc], style='dep', jupyter=True)

We extract the experience by looking to the right for a preposition (e.g. in/with) and then looking for its object and extracting the whole left subtree.

This is obviously quite specific to English.

def extract_adp_experience(doc, label='EXPERIENCE'):
    for tok in doc:
        if tok.lower_ == 'experience':
            for child in tok.rights:
                if child.dep_ == 'prep':
                    for obj in child.children:
                        if obj.dep_ == 'pobj':
                            yield label, obj.left_edge.i, obj.i+1

This works very well! All of our examples are specific.

Notice that:

We’re missing conjugations: we get experience in subsea engineering, but miss “pipelines design” and “construction”
We miss elaborations: “such as Discrete Event Simulation …”
We miss experience in actions (experience in working with children)

show_extraction(examples, extract_adp_experience)

They will need someone who has at least 1015 years of subsea cable engineering experience MISSING

This position is ideally suited to high calibre engineering graduate with significant and appropriate post graduate experience MISSING .

Aerospace industry experience MISSING would be advantageous covering aerostructures and/or aero engines.

A sufficient and appropriate level of building services and controls experience MISSING gained within a client organisation, engineering consultancy or equipment supplier.

Experience in Modelling and Simulation Techniques EXPERIENCE

Any experience of Pioneer or Miser software EXPERIENCE would be an advantage.

For this role, you must have a minimum of 10 years experience in subsea engineering EXPERIENCE , pipelines design or construction.

Has experience within the quality department EXPERIENCE of a related company in a similar role Ideally from a mechanical or manufacturing engineering background.

and have experience of the technical leadership EXPERIENCE of projects to time, quality and cost objectives.

Experience of protection and control design EXPERIENCE at Transmission and Distribution voltages.

Candidates with experience in telesales EXPERIENCE , callcentre, customer service, receptionist or travel are ideal for this role

Experience MISSING dealing with business clients (B2B) would be preferable.

Previous experience MISSING working as a Chef de Partie in a one AA Rosette hotel is needed for the position.

Experience of techniques EXPERIENCE such as Discrete Event Simulation and/or SD modelling Mathematical/scientific background

An alternative strategy would be to look for a phrase like “Experience in/with/using” and then look for the noun phrase

Using spaCy’s noun chunks we can do it backwards (I’m sure there’s an easy way to do it forwards which could be quicker, but it’s nice using spaCy’s noun chunks directly):

def extract_adp_experience_2(doc):
    for np in doc.noun_chunks:
        start_tok = np[0].i
        if start_tok >= 2 and doc[start_tok - 2].lower_ == 'experience' and doc[start_tok - 1].pos_ == 'ADP':
            yield 'EXPERIENCE', start_tok, start_tok + len(np)

show_extraction(examples, extract_adp_experience_2)