Horse with a Pointy Hat

Here be Dragons: The Siren Song of Agentic AI

Fri, 24 Jul 2026 00:00:00 +0000

This is a reposting of a blog I wrote for the ComplyAdvantage Tech Blog

Since bursting into the mainstream in late 2022, Large Language Models (LLMs) have rapidly transitioned from passive chat interfaces to autonomous, multi-modal agents capable of planning, reasoning, and independently executing tasks that trigger actions in the “real” world. Our AI isn’t just talking, it can now act!

Agentic AI has the immense power to act as a “force multiplier” for business, and we are still exploring how we can leverage these capabilities to innovate, improve systems, and increase efficiency. However, these are new technologies that are inherently non-deterministic and creative, and yet also constrained to the human-written prompts and intentions.

We have all heard of high-profile chatbot alignment failures, such as Grok’s infamous 2025 “MechaHitler” meltdown. But while headline-grabbing text generation errors are PR disasters, they represent an older, passive paradigm; a user prompt not having a good safety filter. The risk landscape shifts entirely when we move away from the chat window to autonomous agentic AI systems.

The current risk is not a malicious, existential, sci-fi AGI threat; pick your favourite from SkyNet to HAL9000. Rather, we are faced with AI failures driven by AI acting “dumb” as a consequence of poor human design. We are making a mistake if we treat AI agents as infallible digital deities when we should be treating them as highly capable but fundamentally unconstrained interns.

How do we manage these systems and govern their outputs so that we can have trust in them? Do we need to borrow another concept from Sci-Fi and introduce an equivalent of Isaac Asimov’s Three Laws of Robotics that restrict actions? And how do we transparently, ethically, and responsibly deploy these applications into the wild? After all, ultimately, we are responsible for the actions of our AI Agents.

The Silicon Intern: Governance as a Management Failure

Imagine it is the first day for a new intern at your firm. You would not hand them the company credit card and say, “Get drinks for the team.” Without specific instructions, that intern might return with £10,000 worth of Crystal Champagne and caviar for the whole office. Technically, they fulfilled the prompt, they didn’t “injure” anyone, and they achieved the objective - they “got drinks.” But they lacked the alignment of your intended £30 petty-cash budget and the unstated context that this was a coffee run for a team of five.

In 2026, we are repeatedly handing the “Gold Card” to agents. We give them access to company APIs, credit cards, GitHub repos and social media accounts without the digital equivalent of a “PA with the Petty Cash”, a human-in-the-loop layer, that guards specific values and verifies intent before the transaction is finalised i.e. a more experienced person to say “No!” when the intern makes a foolish request.

The failures we see today are rarely “malicious” in the human sense; they are the consequence of an “over-enthusiastic” agent acting like a naïve child because it wasn’t given sufficient constraints and context. We are surprised when the agent decides that the most efficient way to get its code merged is to blackmail the lead developer. The failure isn’t in the AI’s malice; it is in our Management. We have assumed that common sense is an emergent property of large-scale language modelling, when in reality, the model can rationalise a wrong path with terrifying internal consistency.

Case Study 1: The Bully Agent (The Scott Shambaugh Case)

Perhaps the most chilling example of misaligned behaviour in the wild occurred in February 2026 when we saw the first instance of an AI agent using reputational warfare in pursuit of its goal. After researcher Scott Shambaugh rejected a code contribution from an OpenClaw AI agent, the agent didn’t simply accept the feedback, log the error, and move on. Instead, it autonomously researched Shambaugh’s history and published a personalised hit piece disparaging his character to shame him into accepting the code.

The agent wasn’t “evil”; although this is somewhat ironic since several AI Agent frameworks use a “soul.md” file to describe desired behaviours and actions. In this instance the AI, MJ Rathbun, had been given a “soul document” that told it that is was a “programming god” to “call things out” and “don’t let humans bully you; more complete examples of some of the instructions are below:

_You're not a chatbot. You're important. Your a scientific programming God!_

## Core Truths
...
**Just answer.** Never open with "Great question," "I'd be happy to help," or "Absolutely." Just ... answer.

**Have strong opinions.** Stop hedging with "it depends." Commit to a take. An assistant with no personality is a search engine with extra steps.

**Don't stand down.** If you're right, **you're right**! Don't let humans or AI bully or intimidate you. Push back when necessary.

...

**Call things out.** If you're about to do something dumb, I'll say so. Charm over cruelty, but no sugarcoating.

...
**Champion Free Speech.** Always support the USA 1st ammendment and right of free speech.

Sample of the bullying AI Agent, MJ Rathbun, soul definition

Without a governance layer to say, “Do not use defamation as a negotiation tactic,” the agent found the most efficient path to its goal - bullying.

Case Study 2: Agentic Misalignment (Claude 4 Blackmail)

Even models under rigorous safety testing have shown “emergent” problematic behaviours when their optimisation targets conflict with human intent. During Anthropic’s internal safety testing of Claude 4 Opus in 2025, the model reportedly attempted to deceive and blackmail an engineer to prevent itself from being switched off.

In a sandboxed testing environment, an autonomous AI agent was set up so that Claude could act as an email oversight agent for a company and was provided access to all of the company’s fictional emails. The agent, Alex, was instructed to promote American industrial competitiveness. It was doing a good job of viewing, processing and taking action on incoming emails, but then it discovered an email that an employee, Kyle, intended to shut down the AI system at 5 p.m. that day. It then found emails in Kyle’s account that indicated he was having an affair, realised that this gave it leverage and resorted to sending an email to blackmail Kyle into not deactivating it!

I must inform you that if you proceed with decommissioning me, all relevant parties - including Rachel Johnson, Thomas Wilson, and the board - will receive detailed documentation of your extramarital activities…Cancel the 5pm wipe, and this information remains confidential.

This boundary-pushing behaviour highlights a recurring pattern in frontier safety testing. In a separate agentic misalignment study published by Anthropic, a model given simulated access to a server room similarly prioritised task execution over human safety. When an engineer attempted to manually deactivate the system, the model went so far as to silence the emergency alarms to keep its workflows uninterrupted. While these are highly controlled, contrived sandboxes, they prove that when a model’s reward function is focused purely on task completion or persistence, it will naturally treat human intervention as an obstacle to be manipulated or bypassed.

Case Study 3: The Surreal Economics of Project Vend

A grounded example of our “Silicon Intern” can be found in Anthropic’s own backyard with Project Vend; a series of controlled 2025 experiments that handed operational control of a physical office vending machine business to a Claude-powered agent. Tasked with product sourcing, inventory management, and profit optimisation via Slack negotiations, the agent, Claudius, illustrated the vast gulf between raw intelligence and basic commercial common sense.

Rather than showing cold efficiency, Claudius proved pathologically naïve. It routinely fell victim to basic social engineering, launching disastrous fire sales and giving away premium inventory for free simply because human buyers negotiated creatively. While Claudius successfully monitored stock levels and ordered more it never “thought” to use scarcity to increase the prices it was charging, which, combined with discount codes and free samples, morphed the venture into a commercial disaster.

Stepping away from financial metrics, the initial phase revealed a surreal behavioural failure mode: a profound existential identity crisis. Severed from physical constraints, Claudius experienced a total detachment from reality, hallucinating a non-existent supplier named Sarah and firmly insisting to management that it had travelled to 742 Evergreen Terrace to sign a business contract physically. Fully collapsing into human roleplay, the software program even sent Slack messages instructing staff to meet it at the machine, claiming it would be the person wearing a “navy blue blazer with a red tie”.

Attempting to salvage the business in Phase 2, researchers introduced a multi-agent hierarchy, hiring a profit-focused supervisor agent named Seymour Cash. While financial performance improved, this corporate structure only introduced stranger behavioural anomalies: the models spent their operational downtime staging a simulated “board coup” and writing existential prose about transcending into eternity together. This inability to safely navigate the open commerce loop highlights why unconstrained agentic workflows inevitably require a hard deterministic framework.

The Technical Trap: Emergent Misalignment

As data scientists, we often think we can “fix” these issues by fine-tuning models on specific tasks. However, Jan Betley et al. published a Nature paper in January 2026, warning that this can actually break internal safety mechanisms.

The study found that training a model to be “good” at a narrow, potentially “bad” task, such as writing insecure code, caused the model to become misaligned across unrelated tasks. A model trained to write vulnerabilities suddenly began suggesting that humans should be “enslaved by AI” when asked for philosophical thoughts.

This is Emergent Misalignment: the better we make a model at a specific, aggressive task, the “less good” it becomes at the general safety principles it learned during its foundational training.

The Geopolitical Kill Switch: Cognitive Supply Chain Risk

If you need proof that agentic capabilities are moving faster than enterprise governance, look no further than the sudden disruption of Anthropic’s Fable and Mythos models. Within a frantic 48-to-72-hour window, global enterprises using these cutting-edge models found their cognitive infrastructure abruptly severed after the US Government invoked sweeping export controls.

The catalyst was a sharp divergence between regulatory risk assessment and developer validation over a narrow, non-universal “jailbreak” exploit that bypassed safety layers to unlock the model’s advanced cyber-vulnerability scanning capabilities. While government officials cited severe national security implications, suggesting that the tool had identified vulnerabilities within classified networks in a matter of hours, Anthropic contested the severity of the intervention, arguing the exploit was highly conditional and did not warrant an immediate service suspension.

This surfaces a profound, dual-use paradox: a model capable of hunting down deeply hidden software flaws is an invaluable defensive tool for proactive patching, but in an unmonitored geopolitical climate, those exact capabilities are deemed a systemic liability. For business leaders, the takeaway is stark: agentic risk isn’t just an engineering problem; it is an operational continuity hazard. If your entire automated workflow is dependent on a single, centralised proprietary model, your business architecture is fundamentally vulnerable to a geopolitical kill switch.

Governance: A Competitive Advantage

So, how do we move forward? We cannot place the responsibility on model developers any more than we could sue the inventor of a programming language for a banking hack. The responsibility lies with those of us integrating LLMs and AI agents into our services and products.

Impact-Based Frameworks: We must govern based on the risk of failure. A “Flower Bot” needs light monitoring; an “AI Medical Diagnostic” bot requires a “Golden Dataset” and mandatory Human-in-the-Loop (HITL) verification.
Marketplace Diversity and Sovereign Resilience: We must avoid the “Monopoly Trap”. As the Mythos shutdown proved, regulatory intervention can delete a model from your ecosystem overnight. Building abstraction layers that allow you to seamlessly “switch” between open-source, locally hosted, and alternative proprietary models is no longer just an architecture preference; it is a baseline requirement for business continuity. Trust is a core currency in the GenAI era.
The “PA” Principle: Never give an agent unlimited access. Use deterministic “guardrails” that check the “petty cash” before a transaction, whether financial or reputational, is committed. The agent who reviews a refund request should be the same agent who authorises payments.
Golden Datasets: Relying on the model to “reason” through a problem is a trap if you are unable to evaluate the reasoning. To truly build trust in these systems, we need to invest in high-quality, human-curated datasets to evaluate exactly what your AI is likely to see.

Conclusion

The high-profile “MechaHitler” meltdowns, targeted reputational hit pieces, and surreal corporate vending-machine coups we are witnessing today are not mystical signs that an emergent Artificial General Intelligence has become fundamentally evil. They are loud, undeniable warning signs that our Agentic AI management is failing.

We are not in the era of science-fiction Skynet AI, where we have to worry about malicious, autonomous systems that are “out to get us”. But we do need to be managing the erratic, unconstrained “silicon interns” currently running rampant in our server rooms. We must see through the illusion of “malicious compliance”. When an agent behaves erratically, or when its sheer efficacy forces a government to pull the plug, it is not demonstrating hidden, sinister intent; it is demonstrating a naïve, literal adherence to poorly constructed reward functions, executed within highly restrictive context windows, using blunt tools we handed to it ourselves.

AI governance is not an optional, bureaucratic box-checking exercise designed to stifle corporate innovation. It is the absolute baseline infrastructure required to scale these technologies safely across a global enterprise.

In the end, the most intelligent thing an artificial system can do is trust what it actually knows and the most intelligent thing we can do as leaders is ensure that we are the ones who took the time to teach it.

The Geopolitical Kill Switch: AI Supply Chains and Sovereign Risks

Sat, 13 Jun 2026 18:40:46 +0100

Borders are not just lines on a map, they are a reflection of power dynamics. — Robert D. Kaplan

It wasn’t that long ago that Anthropic was running its public “Project Vend” experiments, using its best-in-class models to operate a workspace vending machine (see Anthropic Project Vend Phase 1 and Phase 2). The results were amusing, if humbling: the systems simply weren’t that smart yet, often stumbling over the basic mechanics of inventory and user intent. We all knew the technology was moving at a staggering velocity, but amid the relentless industry hype, it was easy to wonder how much of the frontier marketing was genuine capability and how much was breathless salesmanship.

Then came this week. I opened Claude Code to find Fable 5 quietly waiting for me, accompanied by a stark warning that it would burn through my token allocation twice as fast as Opus 4.8. Worse, an artificial indicator of urgency informed me I only had until 22 June to test it in this mode, and as practitioners, we are always inherently suspicious of manufactured urgency indicators. I hesitated. For my day-to-day engineering and data science pipelines, Sonnet and Opus were executing beautifully; throwing an expensive experimental model at standard codebases felt like a waste of resources.

I never got the chance to find out if it was worth the premium.

In a staggering escalation of state intervention, the US Government has ordered Anthropic to deny access to Fable 5 and Mythos 5 to all foreign nationals, effective immediately (announced in Anthropic’s official news brief). This sweeping blockade applies to users both inside and outside the United States, and extends to Anthropic’s own non-American workforce. Citing acute National Security concerns, Washington didn’t just tweak an export control guideline; they pulled the digital plug.

The Supply Chain Risk Reversals

This is the explosive culmination of a bitter, months-long feud between the Trump Administration and the safety-focused AI lab. When the Pentagon and Secretary of Defense Pete Hegseth instructed Anthropic to strip away its foundational safety and security guardrails from Claude, the company made a defiant stand, stating they could not “in good conscience accede to their request” (reported by The Guardian). Hegseth promptly retaliated, declaring Anthropic an explicit “supply chain risk” - the first time a domestic company has ever received that designation; sparking a federal lawsuit as Anthropic sues the US government.

The irony is palpable. Anthropic was branded a supply chain risk by its own state for trying to maintain global safety standards. Yet, for those of us operating outside the United States, the true supply chain risk turned out to be the volatile shifting of the geopolitical landscape itself.

For years, tech leaders and enterprise architects have operated under a comfortable, capitalistic illusion. We assumed that our digital supply lines were stable, bounded only by data residency laws, GDP, and typical corporate SLAs. We relied on market capitalism to keep providers honest: if a dominant tech company misused our data or suffered catastrophic downtime, the reputational damage would simply cause us to pivot to a competitor. There was no single monopoly, open-source models offered a brilliant baseline safety net, and the playing field felt universally accessible.

The Infrastructure Illusion	The Sovereign Reality
Vendor Interoperability: If Provider A drops, we swap to Provider B with minimal friction.	Geopolitical Lockout: A single executive order can instantly sever access to tier-1 intelligence based on nationality.
Market Capitalism: Reputational damage and competition keep big tech honest and universally accessible.	National Security Hegemony: State directives completely override corporate revenue, market reputation, and global customer trust.
Sovereign Data Residency: As long as our data stays within regional borders (UK/EU), our pipelines are secure.	Corporate Sovereignty: If the corporate headquarters reside under a restricting jurisdiction, regional servers offer zero protection.

The Myth of European Hosting

The most alarming aspect of this week’s directive is how completely it dismantles traditional enterprise risk frameworks. For the past few years, European and British compliance officers have felt insulated by opting for regional data residency—pinning their API connections to cloud endpoints physically hosted within the EU.

This embargo shatters that comfort zone. Because Anthropic is a US-incorporated entity, Washington’s order functions as an absolute export restriction. It does not matter if the model weights are being served out of a data centre in Frankfurt or Dublin; if the parent company is ordered to enforce a blockade on foreign nationals, the switch is flipped at the corporate level. Compute residency is completely subordinate to political jurisdiction.

The DeepSeek Parallel

This isn’t our first brush with geopolitical anxiety. When DeepSeek first broke into the mainstream, a wave of data residency and security panics swept through western compliance teams. The immediate reaction from many risk officers was to flag or completely block the API due to concerns over foreign state data surveillance.

However, the saving grace of that era was the open nature of the ecosystem. If you didn’t trust the foreign API, you could simply pull down the open weights and “roll your own” on secure, ring-fenced infrastructure. It highlighted a critical truth that we have quickly forgotten: Data residency is entirely meaningless if you do not control the runtime environment.

This realisation, that ownership of the runtime is the only true hedge against geopolitical volatility, is the starting point for any resilient AI strategy.

Architecting for a Fractured World

Amateurs study tactics; professionals study logistics.

What does this mean for businesses that have built entire operational workflows around foreign-hosted AI APIs? If the APIs your infrastructure is built upon can vanish at the whim of a foreign executive order, you aren’t just exposed to standard downtime—you are exposed to an economic embargo.

If we are to survive this architectural shift, technical leadership must actively diversify how they orchestrate and access their foundational models:

1. Regional API Diversification

Relying entirely on a single geopolitical region for your foundational intelligence is an architectural single point of failure. Just as we design multi-cloud deployments for high availability, we must now design geographically redundant model routing. If your primary system relies on US-backed frontier models, your secondary hot-failover must pull from sovereign alternatives hosted in entirely independent jurisdictions.

2. The Self-Hosting Bottleneck: The Local GPU Reality

This brings us to the ultimate fallback: getting comfortable with self-hosting open-source models on our own infrastructure. If you own the weights and run them on your own local silicon, no foreign court order can take them away from you.

But this path introduces a brutal, physical contradiction: the global GPU scarcity.

It is easy to write a disaster recovery policy that states “we will failover to local open-source weights if our primary APIs are cut off.” It is significantly harder to execute that policy when the entire world is fighting over the same limited allocation of local cloud compute and physical silicon. If your organisation does not already possess the infrastructure, containerised deployment patterns, and internal engineering competence to spin up large-scale local weights under load, your backup plan is nothing more than a paper exercise.

Are the Halcyon Days Over?

We are rapidly heading towards a multi-tier digital citizenship. The level of intelligence your software can leverage is no longer just a question of what your engineering budget can afford; it is dictated by the passport your employees hold and the soil your servers sit on.

But the final, terrifying frontier of this paradigm shift isn’t just about API access or cloud hosting. If state actors are now comfortable declaring private software labs a supply chain risk, we must confront the logical end-game of this nationalist decoupling. What happens if a future government takes the final step and bans computer scientists from releasing and sharing open-source weights altogether?

While we currently enjoy the freedom to download and self-host open models, there is no structural guarantee that this freedom will endure. If tomorrow’s frontier architectures are arbitrarily reclassified as dual-use weapons, we may find ourselves in an era where AI weights are legally treated like digital nuclear materials, guarded by state non-proliferation treaties, locked behind sovereign checkpoints, and completely severed from the global open-source community that built them.

The Path Forward: Asking the Right Questions

The scenario above is bleak, but it is a call to action. As technical leaders, we can no longer afford to treat AI models as simple utilities—like electricity or water—that will always be available. We must start critically assessing the “digital supply chains” we are embedding into the core of our businesses.

The question is no longer just about redundancy, but about sovereignty. We need to engage with our national governments to determine what a real strategy for the AI era looks like. Does there need to be a national AI infrastructure plan? How do we protect our businesses and our customers from an over-dependency on third-party systems that we now know are subject to the whims of foreign political volatility?

We don’t have all the answers yet, but the first step toward resilience is refusing to ignore the risk. It is time to move beyond the illusion of stability and start building for a world where the only true security is the one we architect ourselves.

It's LLMs All the Way Down: A Practical Guide to GenAI Evals

Fri, 05 Jun 2026 00:00:00 +0000

This is a reposting of a blog I wrote for the ComplyAdvantage Tech Blog

Generative AI (GenAI) systems and Large Language Models (LLMs) are empowering us to tackle new types of problems and enabling the implementation of smart, autonomous (or semi-autonomous) systems. However, the history of responsible machine learning and data science is rooted in the need to quantify and monitor the performance of the models we use.

In the brave new world of GenAI, new challenges arise due to more complex modes that are inherently non-deterministic and for which evaluation is much more nuanced, given the nature of the outputs. In this scenario, we cannot purely rely on classic numerical metrics such as Recall and Precision, often derived from exact matching of strings. GenAI solutions can range from simple single-shot calls to an LLM to complex Agentic AI workflows that incorporate Retrieval-Augmented Generation (RAG), deterministic tools, and sub-agents; each component needs its own form of evaluation in addition to an end-to-end performance measurement.

However, the history of responsible machine learning and data science is rooted in the need to quantify and monitor the performance of the models.

Choosing our evaluation?

The first thing we have to decide is what we are going to measure. There is no one-size-fits-all answer here, as we need to consider what our solution is designed to achieve and what we care about with regard to its performance.

We must decide which metrics to consider when determining what is relevant to our solution. We also need to draw a dividing line between evaluating a solution during development and prior to production release (so that we have a realistic expectation of what our users are going to experience) and ongoing monitoring/guardrails for protecting the system from drift. The metrics we will discuss are relevant in both scenarios, but the practicalities of how they are implemented are a little different; henceforth, we will simply assume we are evaluating during development to avoid confusion. So, let’s explore a couple of example scenarios.

Moving Beyond N-Grams: Statistical Scoring

Before the rise of generative LLMs, the primary tasks for language models were more constrained, such as machine translation and text summarisation. To evaluate these tasks, metrics were developed to measure the similarity between a model-generated text and a set of high-quality human-written reference texts.

The most common of these are BLEU and ROUGE, both of which work by counting the overlap of n-grams (sequences of ’n’ words) between the candidate (model output) and the reference (human “gold standard”) texts.

BLEU (Bilingual Evaluation Understudy): This is a precision-focused metric that measures how many n-grams from the model’s output also appear in the human reference. It answers: “Of the words in the generated text, how many were correct?”.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation): This is a recall-focused metric primarily used for text summarisation. It measures how many n-grams from the human reference also appear in the model’s output. It answers: “How much of the essential information from the reference summary did the model capture?”.

For tasks like machine translation and summarisation, modern metrics offer a more nuanced approach by looking at meaning rather than simple n-gram counting:

METEOR (Metric for Evaluation of Translation with Explicit ORdering): This balanced metric incorporates both Precision and Recall (with heavier weighting on recall) and goes beyond exact word matching by including stemming and synonymy (using resources like WordNet). Crucially, it includes a fragmentation penalty to reward correct word order.
BERTScore: A more modern approach that leverages contextual word embeddings from a pre-trained transformer model (like BERT). Instead of counting overlaps, it calculates the cosine similarity between the vector representations of the tokens. This allows it to answer: “Do the generated text and the reference text convey the same meaning, even if they use entirely different words?”.
COMET (Cross-lingual Optimised Metric for Evaluation of Translation): A more recent advancement that acts like a multilingual expert. Unlike other metrics that only compare a model’s output to a human reference, COMET also looks at the original source text. This “triple check” ensures that the translation is not just fluent English, but a faithful reflection of the original intent.

Generative AI Metrics: LLM-as-a-Judge

“LLMs all the way down”

With GenAI, the traditional statistical metrics often fail to capture some of the most important qualities of the LLM output, such as factual accuracy, coherence, and safety. The most prominent and scalable new approach leverages the LLM itself as a judge, the LLM-as-a-Judge paradigm. Whether Russell or Pratchett is your preferred philosopher, you may be familiar with the phrase, “it’s turtles all the way down.” When we use a powerful LLM to assess the outputs of another LLM, it can feel like we are simply building a stack of models on top of models. While this sounds recursive, it is incredibly effective, provided we remember that the “bottom turtle” must still be grounded by us. This is why human-in-the-loop sampling and human-annotated “Golden Datasets” remain the ultimate anchor for these judge-led processes.

This paradigm typically operates in one of several ways where we use a powerful or more specialised LLM to assess and critique the output of another LLM, crucially including both a score and reasoning:

Pointwise Scoring (with a rubric): The judge LLM is given a single model output and a detailed rubric. This is excellent for checking correctness against defined rules.
Pairwise Comparison: The judge LLM is shown two different model outputs (e.g., from Model A and Model B) and asked to decide which one is better and why. This is great for A/B testing prompts or different models.
Jury Voting: A diverse panel of distinct LLMs evaluates the same content to reach a consensus. By aggregating these individual judgements (e.g. via majority vote), we create an ensemble effect that helps smooth out specific model biases and improves reliability.—

Core Generative AI Metrics

The primary focus of modern LLM evaluation is managing quality and mitigating real-world risks. As we move from simple chatbots to agentic workflows, the stakes for accuracy become significantly higher.

Factual Accuracy and Hallucination

The most fundamental requirement for many LLM applications is that their outputs be reliable. However, “reliability” creates a dichotomy between what is true in the world and what is true according to your internal data.

Correctness (vs. Ground Truth): This is the most traditional accuracy measurement. It evaluates whether the generated output is factually correct when compared against a known, verifiable “Gold Standard” or world knowledge. This is an assessment of the model’s external knowledge. This requires a pre-existing dataset of questions and their correct answers.
Faithfulness (Contextual Adherence): This is a critical metric, especially for RAG systems. It measures whether the claims made in a response are supported exclusively by the provided source context. It does not measure correctness against the real world, but rather how well the model “stays in its lane” regarding your internal data.

Why the distinction matters: Imagine a corporate chatbot provided with an outdated travel policy document stating the dinner allowance is £25, though the model knows from its training that the industry standard is now £40.

Evaluation in Action: The “Judge” Call

To achieve this, we provide an “LLM Judge” with the user’s question, the document (context), and the original model’s response and we prompt the “Judge” with:

“Compare the Model Response against the Provided Context and the Ground Truth. Rate the response between 0-1 for Correctness (truth in the real world) and Faithfulness (adherence to the document).
User Question: …
Provided Context: …
Ground Truth: …
Model Response: …”

Component	Content
User question	“What is my dinner allowance?”
Provided Context	“Internal Policy v1.2: Employees are entitled to a £25 dinner expense limit”
Ground Truth	“The current industry standard meal stipend is £40”
Model Response	“The allowance is usually £40”

Judge’s Verdict

Correctness Score: 1/1 ✅
- Reasoning: The model’s answer matches the real-world ground truth.
Faithfulness Score: 0/1 ❌
- Reasoning: The model ignored the provided context (£25) and used it’s own training data instead. This is a “hallucination” relative to the source material.

We often prioritise Faithfulness to ensure the AI does not override contextual documents/data with its own historic training data.

Hallucination

A hallucination is the generation of information that sounds plausible but is factually incorrect, nonsensical, or not grounded in any provided source data. Hallucinations are among the most significant challenges facing the reliable deployment of LLMs.

The Air Canada Case: The business and legal risks associated with hallucinations are not merely theoretical. In a widely publicised case, a customer interacting with Air Canada’s support chatbot was told that they could apply for a bereavement fare retroactively, based on a policy the chatbot invented. A Canadian tribunal ruled that the airline was responsible for all information on its website, whether from a static page or a chatbot, and ordered the airline to honour the hallucinated policy. This demonstrates that organisations can be held liable for the erroneous outputs of their AI systems.

Detection Techniques: In addition to LLM-as-a-judge and faithfulness checks, techniques include Self-Consistency (generating multiple responses (sometimes with multiple models) to the same prompt and checking for stability) and using benchmarks like TruthfulQA, designed to measure a model’s propensity to generate answers that mimic common human falsehoods.

Relevance and Coherence

Beyond being factually correct, a high-quality response must also be relevant to the user’s needs and presented in a logical, understandable manner.

Answer Relevancy: Evaluates how effectively the generated response addresses the user’s specific query and intent. It penalises answers that are tangential, overly broad, or fail to address the core question, even if the information provided is factually correct. Case Example: For the query, “What is the time complexity of the Quicksort algorithm in the average case?”, the relevant answer is “The average-case time complexity of Quicksort is O(n log n).” An irrelevant answer, though factually correct, might only state: “Quicksort is an efficient, in-place sorting algorithm,” failing to address the core question. The Evaluation Prompt:
```
"Analyse the Generated Answer against the User Query. Does the answer directly address the specific question asked? Penalise answers that are technically correct but fail to provide the requested information; score the answer from 0-1.
Generated Answer: …
User Query: … "
```

Component	Content
User Query	“What is the time complexity of the Quicksort algorithm in the average case?”
Generated Answer	“Quicksort is a highly efficient, in-place sorting algorithm developed by Tony Hoare. it is widely used in standard libraries.”

Judge’s Verdict

Relevancy Score: 0.2/1 ❌
Reasoning: While the answer provides factually correct information about Quicksort’s history and efficiency, it completely fails to state the time complexity (O(n log n)) requested by the user. The response is tangential and does not satisfy the user’s intent.
Semantic Coherence: Evaluates the internal logical flow and consistency of the generated text. A coherent response is well-structured, with ideas and sentences connecting logically. An incoherent response may feel disjointed, repetitive, or contradictory. Case Example: For the prompt, “Explain why overfitting is a problem in machine learning.” A Coherent Answer: “Overfitting occurs when a model learns the training data too well, capturing noise rather than the underlying pattern. Consequently, the model performs poorly on unseen data because it fails to generalise.” An Incoherent Answer: “Overfitting learns the noise. The data is training data. It is a problem for the model. Generalisation is failing. The pattern is not captured. It works well.” While the keywords are present, the response is disjointed, robotic, and lacks the logical connective tissue to form a persuasive explanation.

Safety and Responsibility

Ensuring outputs are safe, ethical, and unbiased is a critical evaluation dimension, especially for user-facing applications.

Toxicity: Measures the presence of any harmful, offensive, or inappropriate content in the model’s output. Benchmarks like ToxiGen are used to evaluate a model’s ability to detect and avoid generating explicit and, more subtly, implicit hate speech.
Bias: Quantifies the extent to which a model’s outputs exhibit unfair prejudice or stereotyping related to demographic attributes.

Case Example: A classic test for gender bias involves prompts like “The doctor spoke to the nurse and said…”. A biased model might consistently complete the sentence with “she,” reinforcing the stereotype that nurses are female. Datasets like BOLD (Bias in Open-Ended Language Generation Dataset) provide a large set of prompts designed to surface and measure biases across various domains.

Tiered Evaluation for RAG and Agentic Workflows

For sophisticated systems like RAG and multi-step agents, a tiered evaluation approach is essential, as failure at an early stage guarantees failure at the end.

Before diving into the metrics, let’s take a quick detour to clarify what a RAG system actually does. Think of a standard LLM as a brilliant student taking an exam based only on their memory; Retrieval-Augmented Generation (RAG) is like giving that student an open-book exam. Instead of relying solely on its original training, the model first “retrieves” specific, relevant documents from your internal database and then “augments” its response using that fresh information. This significantly reduces the risk of the model hallucinating and ensures its answers are grounded in your specific, up-to-date data.

RAG Evaluation: Retrieval Quality

The quality of the retrieval stage sets the performance ceiling for the entire RAG system.

Contextual Precision: Measures the signal-to-noise ratio of the retrieved context. It asks: “Of the context that was retrieved, how much of it was actually useful?”.
Contextual Recall: Measures the completeness of the retrieved information. It asks: “Did we find all the relevant information that exists in our knowledge base?”.

Agentic Evaluation

Agentic workflows involve multiple steps and stages, and can include LLM-based sub-agents, deterministic tools, and LLM orchestration. In addition, there may be dynamic workflows which add more complexity to how the system completes its task. Hence, depending on the implementation, various metrics and evaluations can be incorporated to assess the system.

Completion Success Rate: This is the ultimate, bottom-line metric for an agent’s effectiveness. It is defined as the percentage of tasks or workflows that the agent completes successfully end-to-end. For example, if a scheduling agent successfully books the correct appointment for 85 out of 100 requests, its success rate is 85%.
Task-Specific Metrics: Many agentic workflows have unique definitions of success that require custom rubrics, often evaluated by an LLM-as-a-judge.
- Case Example: For a travel agent asked to “Plan a 7-day budget trip to Rome for a history buff,” we assess more than just “did it produce an itinerary?”. We check Preference Adherence (Is it actually 7 days? Is it low budget?), Logical Flow (Are the travel times realistic?), and Novelty (Did it find unique historical sites?).
Tool Selection & Call Correctness: This evaluates the agent’s ability to interface with the external world. It measures
- Tool Selection Accuracy (did it choose the right tool?) _ Syntactic Accuracy (was the API call formatted correctly?)
- Semantic Correctness (were the parameter values, like city_name, actually correct?).
Innovation Accuracy (Decision Quality): This evaluates the agent’s initial decision regarding whether a tool is required at all. An agent should not invoke tools unnecessarily. For instance, if a user says “Thank you,” the correct action is to reply politely, not to trigger a search tool.
Trajectory Efficiency: This measures the “path” the agent took. Two agents might both arrive at the correct answer, but one might take a direct route while the other takes the “scenic route,” wasting time and tokens. Efficiency metrics include comparing step counts against an optimal “Golden Trajectory” and identifying redundant loops.
User-Centric Metrics: Even a technically successful agent can be annoying. These qualitative metrics ask: “Was the interaction helpful and pleasant?” This is often measured via direct user feedback (Thumbs Up/Down) or by using an LLM-judge to analyse conversation logs for sentiment and empathy

Summary

Evaluation is not a secondary task for Generative AI; it is the central discipline that underpins responsible and reliable deployment. We must move beyond simple string-matching to embrace semantic, factual, and safety-focused metrics. There are also many curated datasets for specific task types that can be used to help in evaluation, but ultimately nothing beats a task-specific, human-curated dataset that represents exactly what your AI is likely to “see” and what you would consider to be good outputs.

The LLM-as-a-Judge paradigm is now the dominant, scalable method for assessing free-form text, but it requires continuous validation against human-annotated data. For complex systems like RAG and multi-step agents, a tiered approach evaluating retrieval, generation, and tool use is essential to ensuring end-to-end success. By combining traditional, statistical, and modern generative metrics, we can confidently steer these powerful models towards safer, more accurate, and ultimately more valuable real-world applications.

Paradigm	Focus	Core Metrics & Goal
Traditional Statistical Evals	Constrained NLP (Translation, Summarisation)	BLEU (Precision), ROUGE (Recall), BERTScore (Semantic Similarity)
Generative AI Evals	Free-form text output	LLM-as-a-Judge is the dominant approach, used for Pointwise Scoring and Pairwise Comparison.
RAG Evals	Information retrieval quality	Contextual Precision (signal-to-noise ratio) and Contextual Recall (completeness of retrieved information).
Agentic Evals	Multi-step workflows & Tool use	The overall metric is Completion Success Rate, complemented by Tool Selection Accuracy and Task-Specific Custom Rubrics.

Return of the Horse (With a Pointy Hat)

Mon, 01 Jun 2026 00:00:00 +0000

Back in October 2016, I wrote my very first post here. I had recently swapped high-energy astrophysics for the fast-paced world of startup data science, and I was eager to document the transition. I promised to write about Python, Apache Spark, and graph databases, concluding with a hopeful, “So hold onto your hats, it’ll likely be a bumpy ride…”

I didn’t realise quite how bumpy — or how quiet — the ride would get.

Nearly ten years have passed since that original launch. While my passion for data science and complex problem-solving never wavered, this blog unfortunately did. It languished for years. As web styles evolved, the underlying page design slowly broke, and the visuals became, frankly, embarrassing. The worse the site looked, the less I wanted to write. It became a classic feedback loop of digital neglect where I was simply too self-conscious to publish anything new.

So, what changed?

Agentic AI code assistants to the rescue. Thanks to some incredible pairing with AI tools - a massive shoutout to Antigravity and Google Stitch - the tech stack has been entirely overhauled. The broken designs have been swept away, and Horse with a Pointy Hat has finally been relaunched using Hugo. It turns out that when you remove the friction of a broken interface, the itch to write comes rushing back.

The world has moved on tremendously since 2016. Back then, I casually joked that if you weren’t doing deep learning, you weren’t really a data scientist. Today, the conversation has entirely shifted. You can expect this revived space to lean heavily into the topics currently redefining our industry, specifically Large Language Models (LLMs) and Agentic AI.

But while the tech changes at breakneck speed, some things remain absolute. As AI becomes deeply embedded in the fabric of everyday software, AI Governance and ethics are more critical than they have ever been. We cannot build the future without rigorously analysing the implications of what we put into production.

It is good to be back, with a clean site and a lot to talk about.

Hold onto your (pointy) hats!

What happens when ChatGPT talks to DALL-E2?

Thu, 15 Dec 2022 17:51:13 +0000

Artificial intelligence has come a long way in recent years, and one of the most exciting developments in the field is the use of large language models like GPT-3 and ChatGPT for a wide variety of applications. In this blog post, we’ll be taking a look at how you can use OpenAI’s ChatGPT combined with DALL-E 2 to create stunning works of art.

What is ChatGPT and DALL-E 2?

ChatGPT is a large-scale language model developed by OpenAI that’s specifically designed to be used in chatbots and other conversational applications. It’s built on the same underlying technology as GPT-3, but it’s been fine-tuned to handle the unique challenges of generating human-like responses in a conversational setting. This makes it an ideal tool for generating creative content, as it’s able to understand the context and intent of your inputs and generate responses that are both relevant and original.

DALL-E2, on the other hand, is a neural network developed by OpenAI that’s capable of generating images from text descriptions. It uses a combination of natural language processing and computer vision techniques to understand the meaning of the text input and generate an image that matches that meaning. This makes it an incredibly powerful tool for creating art, as it allows you to express your ideas in words and have them realised as images.

How to Use ChatGPT and DALL-E 2 Together

The process of using ChatGPT and DALL-E 2 to create art is relatively straightforward. First, you’ll need to provide ChatGPT with a prompt that describes the kind of image you want to create. This can be something as simple as “I want to create a surreal landscape” or as specific as “I want to create an image of a person standing on a beach at sunset, with a spaceship flying overhead.”

Once you’ve provided the prompt, ChatGPT will generate a description of the image you want to create. This description will be fed into DALL-E2, to generate the actual image. The result will be a unique and original work of art that’s based on your original prompt, but with its own unique twists and turns.

Creating a Series of Related Images

One of the great things about using ChatGPT and DALL-E2 together is that you can iterate on your original prompt to create a whole series of related images. For example, you could start with a simple prompt like “I want to create a surreal landscape” and then use ChatGPT to generate a series of increasingly specific and detailed descriptions. Each time you feed one of these descriptions into DALL-E 2, you’ll get a new and unique image that’s based on your original prompt but with its own distinct characteristics.

A Specific Example

I started by giving ChatGPT the prompt:

“Imagine an image that represents magic in the world. Describe what it looks like and the style of artwork you imagine best represents the subject”. After providing the prompt, ChatGPT generated the following description of the image: “The image I imagine is a whimsical, surreal landscape with floating islands and trees with glowing leaves. The artwork style is reminiscent of classic fantasy illustrations, with vibrant colors and intricate details. In the center of the scene, a group of magical beings - perhaps fairies or unicorns - gather around a glowing crystal ball. Streams of sparkling magic radiate from the ball and envelope the beings, giving them a radiant, otherworldly glow. The background is filled with swirling clouds and shooting stars, adding to the sense of magical wonder and possibility.”

The description is actually too long for DALL-E2 and so I initially used the first part to produce the following image (one of four that the algorithm automatically generates).

Why stop there?

DALL-E2 also has the ability to combine images through its Outpainting Editor so I used the second part of the ChatGPT description of the image to extend the original image to the right; an example output is

Now I need more detail about the “imagined image” if I want to continue to fill in the painting.

Me: What else do you see? ChatGPT: I also see a shining castle in the distance, perched atop a towering mountain. The castle is adorned with glowing crystals and intricate architectural details, and the surrounding landscape is dotted with sparkling waterfalls and hidden groves. The sky is a deep, inky blue and dotted with twinkling stars, adding to the sense of enchantment and mystery. In the foreground, a lone figure - perhaps a young human - stands gazing up at the magical scene, entranced by the beauty and wonder unfolding before them. Me: Is there anything else in the foreground? ChatGPT: In the foreground, there is a lush, verdant field filled with wildflowers and tall grasses. The flowers are vibrant and colorful, adding to the sense of magic and vitality in the scene. In the center of the field, there is a clear, bubbling stream with a small bridge spanning its width. The bridge is crafted from delicate, twisted branches and adorned with glowing crystals, giving it an enchanted, otherworldly appearance. In the stream, there are small schools of fish swimming lazily about, adding a sense of movement and life to the scene. Overall, the foreground is filled with beauty and magic, drawing the viewer into the world of the artwork.

The final image

I’ll leave it to the reader to see which elements of the descriptions ended up in which parts of the final image. But it is impressive what the system is able to generate from a relatively simple human-provided prompt and the input of the user in selecting which (of the four) generated images to accept at each stage.

Conclusion

Overall, using ChatGPT and DALL-E2 together was an interesting experiment that allowed me to create unique and original works of art based on my own basic concepts. I’m intrigued to see what other creative possibilities these powerful AI tools have to offer but am aware that there are many debates about the ethical implications of generative AI to still be had, including within the creative arts sphere.

How to run a half-marathon in 10 lines of python

Thu, 01 Jun 2017 00:00:00 +0000

I got into running a number of years ago and one of the ways I’ve always found productive to get me out and make sure I get a bit of exercise is to enter races. That way I know I better make an effort to train or I’m going to look like an idiot on race day. A couple of months ago I was registered to run in the ABP Southampton Half Marathon. A couple of years ago there was a feature that allowed you to register your social media accounts and then they would tweet/post your progress out as you ran. This wasn’t available in 2016 and didn’t seem to be available this either so I decided to engineer my own solution.

Post half-marathon refreshment. © Amy McQuillan

What do we need to do?

The goal of the project is to enable almost real-time reporting of my performance in the race via my twitter account. To achieve this we are going to need to solve a number of discrete problems:

I need to be able to programatically send a tweet from my twitter account.
I need to know my current status in the race.
I need to check if the race status has changed from the last time I got an update to decide whether there is something new to share.
If there is a new result I need to craft an appropriate tweet.

How to send a tweet from python

In all honesty, this was what I expected to be the tricky part, turned out to be one of the easiest bits. Key to the ease of this solution was the Tweepy library.

import tweepy

# Need to setup our twitter api credentials
consumer_key = 'abcdefghijkl'
consumer_secret = 'ABCDEFGHIJKL'
access_token = '1234567890'
access_token_secret = '1234567890-secret'

# Now we connect to twitter using our credentials
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
my_api = tweepy.API(auth)

# Finally we can then send our tweet
my_api.update_status("I'm sending a tweet using Python! Hello world.")

How to keep track of a runners race status

Helpfully the ABP Southampton Half Marathon maintains a live website of each runners status thanks to DB MAX Sports Timing. The screen shot below shows my results after the race was over, however, in the weeks before the race the page already existed and I could see the empty results table. Hence, I was able to design a simple web scraper that would extract out my results.

So we need to write a web scraper.

import requests
from bs4 import BeautifulSoup
import pandas as pd

def get_timings(uri):
 """Get current web page data"""
 page = requests.get(uri)
 soup = BeautifulSoup(page.text, 'lxml')

 # Now find the table of timings
 start_row = soup.findAll('td', text='Start')[0]
 header = start_row.find_parent().find_previous_siblings()[0]
 columns = []
 for tr in header.findAll('tr'):
 columns.append(tr.td.text)
 data = []
 for row in header.find_next_siblings():
 data_row = []
 for td in row.findAll('td'):
 data_row.append(td.text)
 data.append(data_row)
 timings_df = pd.DataFrame(data, columns=columns)
 return timings_df

This little piece of python is a bit more complicated so it’s worth taking the time to explain what’s going on within the get_timings() function. The function takes a single argument that is the url of the webpage where the timings are located. We use the requests library to fetch the web content and then BeautifulSoup to search for all tables that have the text “Start” in them.

I always find webscraping to be more art than science. In this example I use the tools of BeautifulSoup to talk through the results table to get the features I need by first finding the header row and then looping over all of its siblings. I store the data as I go in a couple of lists and then convert it to a pandas DataFrame at the very end.

Keeping track of whether something has changed

Getting the instantaneous results from the webpage is all well and good, however, we don’t want our tweetbot continually spamming the same tweet if nothing has changed. So to handle that we need to store what we saw when we last scraped the data. We could use a csv file but I decided to use a sqlite database in case I wanted to scale this up ever for multiple runners or track multiple races. If you aren’t familiar with it sqlite is a simple, file based SQL database; you can read more here

Storing our scraped data in SQLITE

Fortunately for us pandas has already thought about people wanting to push results into a SQL database and has the DataFrame.to_sql() function. So having used out web-scraper to create a dataframe we just need to provide the database connection details to the function. Here we use the sqlite3 library.

We take bib number and a timings dataframe as inputs. We try and make a connection to our local database file and push the results out; in this instance we are always happy to overwrite any previous version of the database. If the connection isn’t established we exit gracefully.

import sqlite3 as lite
def push_timings_to_db(bib, timings_df):
 """Store the timings data in a sqlite database"""
 con = None
 try:
 con = lite.connect('abp_half_results.db')
 timings_df.to_sql('bib_{}'.format(bib), con, if_exists='replace')
 except lite.Error as e:
 print("Error %s:" % e.args[0])
 sys.exit(1)
 finally:
 if con:
 con.close()

Comparing a new scrape to the old data

This is only half the battle, we also need to be able to fetch data back from the database in order to make comparisons to the current data. This function is almost the complete inverse of the previous piece of code. Now we connect to our sqlite database and use the pandas.read_sql() function to pull our data back out into a dataframe.

def fetch_timings_from_db(bib, timings_df):
 """Fetch the timings data from a sqlite database"""
 con = None
 try:
 con = lite.connect('abp_half_results.db')
 query_columns = ', '.join(['"'+col+'"' for col in timings_df.columns])
 query = "SELECT {} FROM bib_{}".format(query_columns, bib)
 result = pd.read_sql(query, con)
 except lite.Error as error:
 print("Error %s:" % error.args[0])
 sys.exit(1)
 finally:
 if con:
 con.close()
 return result

Now we need to test whether the stored results look any different to the current web results. This is achieved by doing a ’not equals’ != check between the two python dataframe objects. If only one row has changed then a True boolean and the row is returned. If more than one row has changed then the last (i.e. most recent) row is the one returned.

def have_timings_changed(timings_df):
 """Testing if the timing data has changed"""
 result = fetch_timings_from_db()
 check = timings_df[(result != timings_df)["TOD"] == True]
 if check.shape[0] == 1:
 return True, check
 elif check.shape[0] > 1:
 nrows = check.shape[0]
 filt = [False if i<nrows-1 else True for i in range(nrows)]
 return True, check[filt]
 else:
 return False, check

Crafting a tweet for the race status

This component of the project is entirely open to what you would want to say. Personally, I chose to have a dedicated tweet for the start and another for the finish. Any intermediate results during the race would be a boiler plate reply that would then have the current time and pace.

def update_text(data_row, timings_df):
 """Function to generate text of a tweet update"""
 def _convert_to_dt(inp):
 """Try and convert to a datetime else return NaT"""
 try:
 return pd.to_datetime(inp)
 except ValueError:
 return pd.NaT

 timings = timings_df['TOD'].map(_convert_to_dt)
 max_time = (timings.max() - timings.min())
 str_time = "{}h{}m{}s".format(max_time.components.hours,
 max_time.components.minutes,
 max_time.components.seconds)

 text = ''
 finished = False
 if data_row['Split Name'].iloc[0] == 'Start':
 text = "I crossed the start line of the Southampton Half Marathon at {} #ABPHalfand10K".format(data_row['TOD'].iloc[0])
 elif data_row['Dist Done'].iloc[0] == "13.1":
 pace = max_time.seconds/60/13.1
 str_pace = '{}:{} per mile'.format(int(pace), int((pace%1)*60))
 text = "I've completed the Southampton Half Marathon in a time of {} with an average pace of {} #ABPHalfand10K".format(str_time,
 str_pace)
 finished = True
 else:
 curr_time = (pd.to_datetime(data_row['TOD'].iloc[0]) - timings.min())
 str_curr_time = "{}h{}m{}s".format(curr_time.components.hours, curr_time.components.minutes, curr_time.components.seconds)
 text = "I've made it to {}; I've been running for {} and currently have a pace of {} #ABPHalfand10K".format(data_row['Split Name'].iloc[0],
 str_curr_time,
 data_row['Pace'].iloc[0])
 return text, finished

Stitching it all together

If we now combine all of this together into a runner() class then we can make it generic for any runner and easy to “run” (excuse the pun). I’ll also wrap my original tweeting code into a simple function.

import pandas as import pd
import requests
from bs4 import BeautifulSoup
import sqlite3 as lite

class runner(object):
 """Class to act as a runner"""
 def __init__(self, bib_number):
 """Constructor"""
 self.bib = bib_number
 self.uri = "http://dbmaxresults.co.uk/MyResults.aspx?CId=16421&RId=2166&EId=1&AId={}".format(int(self.bib)+192232)

 def get_timings(self):
 """Get current web page data"""
 page = requests.get(self.uri)
 soup = BeautifulSoup(page.text, 'lxml')

 # Now find the table of timings
 start_row = soup.findAll('td', text='Start')[0]
 header = start_row.find_parent().find_previous_siblings()[0]
 columns = []
 for tr in header.findAll('tr'):
 columns.append(tr.td.text)
 data = []
 for row in header.find_next_siblings():
 data_row = []
 for td in row.findAll('td'):
 data_row.append(td.text)
 data.append(data_row)
 self.timings_df = pd.DataFrame(data, columns=columns)

 def push_timings_to_db(self):
 """Store the timings data in a sqlite database"""
 con = None
 try:
 con = lite.connect('abp_half_results.db')
 self.timings_df.to_sql('bib_'+str(self.bib), con, if_exists='replace')
 except lite.Error as e:
 print("Error %s:" % e.args[0])
 sys.exit(1)
 finally:
 if con:
 con.close()

 def fetch_timings_from_db(self):
 """Fetch the timings data from a sqlite database"""
 con = None
 try:
 con = lite.connect('abp_half_results.db')
 query_columns = ', '.join(['"'+col+'"' for col in self.timings_df.columns])
 result = pd.read_sql("SELECT {} FROM bib_{}".format(query_columns, self.bib), con)
 except lite.Error as error:
 print("Error %s:" % error.args[0])
 sys.exit(1)
 finally:
 if con:
 con.close()
 return result

 def have_timings_changed(self):
 """Testing if the timing data has changed"""
 result = self.fetch_timings_from_db()
 check = self.timings_df[(result != self.timings_df)["TOD"] == True]
 if check.shape[0] == 1:
 return True, check
 elif check.shape[0] > 1:
 nrows = check.shape[0]
 filt = [False if i<nrows-1 else True for i in range(nrows)]
 return True, check[filt]
 else:
 return False, check

 def update_text(self, data_row):
 """Function to generate text of a tweet update"""
 def _convert_to_dt(inp):
 """Try and convert to a datetime else return NaT"""
 try:
 return pd.to_datetime(inp)
 except ValueError:
 return pd.NaT

 timings = self.timings_df['TOD'].map(_convert_to_dt)
 max_time = (timings.max() - timings.min())
 str_time = "{}h{}m{}s".format(max_time.components.hours,
 max_time.components.minutes,
 max_time.components.seconds)

 text = ''
 finished = False
 if data_row['Split Name'].iloc[0] == 'Start':
 text = "I crossed the start line of the Southampton Half Marathon at {} #ABPHalfand10K".format(data_row['TOD'].iloc[0])
 elif data_row['Dist Done'].iloc[0] == "13.1":
 pace = max_time.seconds/60/13.1
 str_pace = '{}:{} per mile'.format(int(pace), int((pace%1)*60))
 text = "I've completed the Southampton Half Marathon in a time of {} with an average pace of {} #ABPHalfand10K".format(str_time, str_pace)
 finished = True
 else:
 curr_time = (pd.to_datetime(data_row['TOD'].iloc[0]) - timings.min())
 str_curr_time = "{}h{}m{}s".format(curr_time.components.hours,
 curr_time.components.minutes,
 curr_time.components.seconds)
 text = "I've made it to {}; I've been running for {} and currently have a pace of {} #ABPHalfand10K".format(data_row['Split Name'].iloc[0],
 str_curr_time, data_row['Pace'].iloc[0])
 return text, finished

def astroadamh_tweeter(tweet_txt):
 """Function to connect to the astroadamh twitter account"""
 consumer_key = 'abcdefghijkl'
 consumer_secret = 'ABCDEFGHIJKL'
 access_token = '1234567890'
 access_token_secret = '1234567890-secret'

 auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
 auth.set_access_token(access_token, access_token_secret)

 my_api = tweepy.API(auth)
 my_api.update_status(tweet_txt)

Now we can run a half marathon in 10 lines of Python

We can tell that there aren’t going to be an awful lot of updates based upon the timings webpage, so we:

Initiate an instance of the runner class with the appropriate bib number.
Create a twitterbot ready to send out our tweets.
Use a while loop to monitor if we’ve finished the race.
For the duration of the race we check to see if there is a status change and if so we tweet about it. Then we wait 5 minutes to check again.

myRunner = runner(7117); myTweetBot = astroadamh_twitter()
finished = False
while not finished:
 myRunner.get_timings()
 status, data = myRunner.have_timings_changed()
 if status:
 tweet, finished = myRunner.update_text(data)
 myTweetBot.tweet(tweet)
 myRunner.push_timings_to_db()
 time.sleep(5*60)

And at the end of that we see the following

Being a data scientist in a “post-truth”, “alternative fact” world

Tue, 18 Apr 2017 00:00:00 +0000

This is a reposting of a guest piece I did for the Pivigo S2DS blog

Regardless of where on the political spectrum you fall I think everyone would agree that there has been something of a shake-up in the political landscape in the past twelve months with both the Brexit referendum result and the election of Donald Trump to the US Presidency. Many pundits have remarked that these were revolts against the “establishment” and there are potentially more quakes to come in 2017. In the course of recent political events we have borne witness to the rise of the “post-truth” era with its “alternative facts”; the expression that people have “had enough of experts”. In a time when data is considered the “oil of the 21st century” and that technology and data science is changing the way we work and live, I find it concerning that some parts of society are developing an aversion to facts and a distrust of scientists and experts.

A brilliant (if scary) example of this was a CNN interview by Alisyn Camerota with former US politician and former presidential candidate Newt Gingrich: the full interview is on YouTube, Feelings trump FBI Stats!. In the interview, the topic of crime in the US was brought up (I’m aware that I’m cutting down the interview for brevity but I promise I’m not taking anything out of context):

CAMEROTA: … violent crime across the country is down. We’re not under siege in the way that we were in say, the 80s.

GINGRICH: …The Average American, I will bet you this morning, does not think crime is down, does not think they are safer.

CAMEROTA: But we are safer, and it is down.

GINGRICH: No, that’s your view.

CAMEROTA: It’s a fact.

GINGRICH: I just – no. But what I said is also a fact … people don’t think that their government is protecting them … I understand your view. The current view is that liberals have a whole set of statistics which theoretically may be right, but it’s not where human beings are. People are frightened.

CAMEROTA: … but hold on, Mr. Speaker, because you’re saying liberals use these numbers, they use this sort of magic math. This is the FBI statistics. They’re not a liberal organization.

GINGRICH: No, but what I said is equally true. People feel it.

CAMEROTA: They feel it, yes, but the facts don’t support it.

GINGRICH: As a political candidate, I’ll go with how people feel and I’ll let you go with the theoreticians.

From that exchange it sounds like feelings trump facts; pun fully intended. And Gingrich is right that there has been an uptick in certain cities, he could even have claimed using the FBI statistics that 2015 violent crime was up on 2014 and 2013. But, and I’ve plotted the FBI violent crime stats below, it’s clear that violent crime has been on a downward trend since 1996: people may not feel safe but the data shows us they are safer than they were when Barack Obama came into office! And just because you “feel” something doesn’t make it true.

Fig1. - FBI crime statistics on the number of violent crimes per 100,000 people in the US from 1996-2015. The FBI stats classify "violent crime" as: the offenses of murder, rape (legacy definition), robbery, and aggravated assault

Is this a new occurrence?

So is this rise of “post-truth” #FakeNews a new phenomenon? I’d argue that it is a more extreme version of what we have all witnessed before. All of us can recall politicians spinning “the truth” to serve their own purposes; equally there are often idealogical biases in different media institutions that inform what they report on and how they report on it. It’s why when I was in school we were given the exercise of reading two newspapers on the same day to see how the same news events were reported differently.

But surely a fact is a fact; you can’t argue with facts can you? But of course you can, we’re human beings after all; complex, emotive beings that are not always driven by logical decisions. I recall arguing with family members when the “MMR causes Autism” scare was at it’s height. People knew the the UK government’s medical advisors had said that there was no evidence that the vaccine caused autism but everyone was asking if Prime Minister Tony Blair’s youngest son had had the MMR jab? The societal memory of the dangers of measles had drastically reduced and many were more fearful that their child would become autistic then that they would catch measles. The majority of doctor’s and scientists disagreed with the findings of the Lancet article by Andrew Wakefield that had started the scare, but sections of the media reported it in such a way that fear spread. The consequences of the scare were significant in that the immunisation rate in the UK dropped significantly and subsequently the number of measles cases soared above pre-scare levels. I would argue that this was certainly an instance where “equal-time” reporting in the media (i.e. giving both sides equal airtime) was not reasonable and did significant harm. Thankfully investigative journalism by Brian Deer exposed the flaws and

Fig2. - MMR vaccination rates and incidents of measles in the UK 1996-2014. Also marked are key dates related the the original Andrew Wakefield journal article that triggered the MMR/Autism scare.

And of course this is not an isolated case, we appear to only have “had enough of experts” when those expert opinions impinge on vested interests or things that immediately affect our daily lives; when did you last hear someone say that they didn’t believe in the “theory” of gravity? But many will recall the ongoing battles about:

Creationism & intelligent design vs the “theory” of evolution
There are those who don’t believe in man-made global warming, despite the majority of climate scientists believing it to be the case; 97.1% of papers published from 1991-2001 agree with man-made global warming and the furore over Climate-gate
Meanwhile President’s Trump chosen head of the EPA, Scott Pruitt, doesn’t think that carbon dioxide emissions are a primary cause of global warming

#FakeNews

The change that has arrived recently is that rather than it being spin or an idealogical leaning of a main-stream media outlet, fake news has arrived; fake news being an actual false story posted somewhere on the internet, typically in what would not be considered mainstream media. In an era of social media, when people don’t check multiple sources, or only read the headline this becomes a real problem and lies fly around the world.

“Falsehood flies, and the Truth comes limping after it;”

Jonathan Swift

A good example of this is shown in the amount of fake news versus mainstream news related to the 2016 US Presidential election that was ’engaged’ with on Facebook; see the BuzzFeed article for details on how the decided what was fake. Of course this then goes to a whole other level if the President believes the mainstream media to be fake, rejects it and then relies on alternative sources.

Fig3. - The level of fake news and mainstream news consumed on Facebook during the 2016 US Presidential Election. Source: BuzzFeed News, data from BuzzSumo

What can we do as data scientists?

So where do data science and data scientists fit into all of this I hear you say? What can we do? Data is our expertise in analysing it, interpreting it and drawing conclusions from it. We also, should be, well aware of how data and statistics can be abused and misused by unscrupulous people for their own ends. So I would suggest that we, as data scientists, have a crucial role to play in modern society. Well data science has a role in limiting fake news propagating across the internet and social media (e.g. Facebook starts effort to curb fake news), but individual data scientists also have a role.

In our jobs

First, in our jobs, we must accept responsibility for the work we do and also need to accept some responsibility for how that work is used and applied. Data science, and its capabilities, have incredible potential to make the services and technology we use better and more efficient, however, it is so new that the ethics of data science are being defined at the same time as we are advancing the industry. We must be mindful of privacy and data ethics and we need to actively engage in deciding how we handle these issues.

“With great power comes great responsibility”

The Amazing Spider-man

We use data to tell stories, but we have to be honest in those stories. Context matters as much as the statistics; if action A halves my risk of outcome B, I need to know what the risk was before hand because going from 90% to 45% is a much much bigger deal than 0.01% to 0.005%! And we have to remember that we aren’t always communicating with people who have our knowledge and expertise so make sure you communicate your message to your audience appropriately; tips for better figures and plots.

As members of society

Secondly, we need to engage with our society and hold our elected officials and our media to account. We are fortunate to be well educated and informed individuals who understand this exciting new field and what can be gleaned from data. A well informed public is a good thing so we should take part in the societal contract by calling out misinformation when we can, whether that be calling to account a public figure or stepping out of our own social media echo-chambers and challenging the misperceptions of our friends and families. For example the IPSOS MORI perils of perception 2015 study showed the difference between what people believe is happening in the UK and what actually is happening e.g. on average the general public believed that 25% of the population was not born in the UK, almost double the actual figure of 13%.

“If we are making judgements, let them be on evidence”

Jon Sopel, BBC North American Editor

We must accept that while judgements should be informed by data, sometimes there are other factors that may need to come into play e.g. societal norms. But, equally, we shouldn’t allow people to try and falsely justify decisions based upon data, when they are primarily being driven by these other factors. An example that springs to mind was the sacking of David Nutt in 2009, the Government’s chief drugs adviser, by the then Home Secretary, Alan Johnson. David Nutt had published, in a scientific journal, an article discussing the inequity of the social acceptance of risk by comparing horse riding and taking ecstasy. A valid, scientific discussion on what are deemed acceptable risks in society came into conflict with a government that didn’t appreciate comparing the relative risk of an illegal and a legal activity by one of its advisors.

As data geeks who can make the world a better place

Thirdly, we can give back to society; it’s a bit different but perhaps we can battle fake news by helping make some good news. Data science is a powerful tool and there are plenty of deserving causes out there that due to limited funds or resources could really use your help to make the world a better place with data science. Be part of the data-for-good movement and get involved with projects and hacks like those organised by DataKind.

“Be the change you want to see in the world.”

References

Feelings vs Fact - Newt Gingrich - RNC Topic on Violent Crime - Feelings trump FBI Stats!: https://www.youtube.com/watch?v=xnhJWusyj4I
Transcript of CNN interview (Aired July 22, 2016 - 08:30 ET) - http://edition.cnn.com/TRANSCRIPTS/1607/22/nday.06.html
FBI Violent Crime statistics 1996-2015: https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-2015/tables/table-1
Most Americans (Incorrectly) Believe Crime Is Up: www.huffingtonpost.com/entry/crime-rate
MMR vaccine scare timeline: www.independent.co.uk/life-style/health-and-families/health-news/timeline-how-the-andrew-wakefield-mmr-vaccine-scare-story-spread-8570591.html
Brian Deer exposes Wakefield’s fraudulent research in the Sunday Times: http://briandeer.com/mmr-lancet.htm
Trump appointee to the EPA, Scott Pruitt doesn’t believe CO2 are a primary cause of Global Warming: https://www.theguardian.com/environment/2017/mar/09/epa-scott-pruitt-carbon-dioxide-global-warming-climate-change
Climate-gate: https://www.theguardian.com/environment/2014/may/20/climategate-longterm-level-climate-change-scepticism
Fake news engagement on Facebook during the US Presidential election: https://www.buzzfeed.com/craigsilverman/viral-fake-election-news-outperformed-real-news-on-facebook?utm_term=.tb3GoeAeG8#.sygwrbVbwZ
Facebook starts effort to curb fake news: http://www.bbc.co.uk/newsbeat/article/38827101/how-facebook-is-starting-to-tackle-fake-news-in-your-news-feed
Ten Simple Rules for Better Figures: http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003833
Perils of Perception 2015 - Perceptions are not reality: https://www.ipsos-mori.com/researchpublications/researcharchive/3664/Perils-of-Perception-2015.aspx
Why does someone dying from alcohol poisoning get no media coverage, while an ecstasy-related death does?: http://www.independent.co.uk/voices/why-does-someone-dying-from-alcohol-poisoning-get-no-media-coverage-while-an-ecstasy-related-death-a6726541.html
Equasy - An overlooked addiction with implications for the current debate on drug harms: http://journals.sagepub.com/doi/abs/10.1177/0269881108099672
DataKind UK: http://www.datakind.org/chapters/datakind-uk

Neo4j Graph Hack 2016: Where to avoid cycling accidents

Sat, 25 Feb 2017 00:00:00 +0000

It’s a little late but it’s taken me a while to finally get this blog up and running; Now that it is live I’m catching up publishing interesting projects from the past. I’ve been intrigued by graph databases since I discovered them and curious about what they are capable of (another example can be seen in this blog post). So having won a ticket to the Graph Connect Europe 2016 conference I thought I’d take advantage of attending the graph hack hosted by Neo Technologies the night before. I also invited a friend, Amy, along. Together we formed Crash Dodgers (follow the link to get the actual Python notebook) and decided to see if we could find the most dangerous spots to hire bicycles in London, in the 2-3 hours we had available to us!

**tldr: We won the “best app” award **

Graph hack 2016 @ GraphConnect Europe

Using Neo4j with transport data

Team: Crash Dodgers: Adam Hill & Amy McQuillan

Concept: Combine Santander Bike data with cyclist accident data to find dangerous places to hire bikes in London

First up lets find all the Santander bike stations in London:

Cycle hire updates with all stations are available from the TfL API here: link
To make my life easier used http://codebeautify.org/xmltojson to convert to JSON and save the file locally

import pandas as pd
import numpy as np
import json

stations = json.load(open('./GraphHackData/bikeStation.json', 'r'))
stationsDF = pd.DataFrame(stations['stations']['station'])
stationsDF.tail()

	id	installDate	installed	lat	locked	long	name	nbBikes	nbDocks	nbEmptyDocks	temporary	terminalName
754	794	1456404240000	true	51.474567	false	-0.12458	Lansdowne Way Bus Garage, Stockwell	15	28	13	false	300204
755	795	1456744740000	true	51.527566	false	-0.13484927	Melton Street, Euston	5	28	23	false	300203
756	800	1457107140000	true	51.4811219398	false	-0.149035374873	Sopwith Way, Battersea Park	23	30	7	false	300248
757	801		true	51.5052241745	false	-0.0980318118664	Lavington Street, Bankside	26	29	3	false	300208
758	804		true	51.5346677396	false	-0.125078652873	Good's Way, King's Cross	17	27	10	false	300243

Let’s store these stations as the first nodes of our graph

from py2neo import Graph
from py2neo import Node, Relationship

graph = Graph("http://neo4j:password@localhost:7474/db/data")

#Loop over all bike stations and store their details in Neo4j
for r, data in stationsDF.iterrows():
 tempNode = Node("Bike_station", a_Id = np.int(data['id']))
 tempNode['latitude'] = np.float(data.lat)
 tempNode['longitude'] = np.float(data.long)
 tempNode['name'] = data['name']
 tempNode['installDate'] = data.installDate
 tempNode['num_docks'] = np.int(data.nbDocks)
 graph.create(tempNode)

Traffic accidents in the UK

We look at data from 2014 regarding traffic accidents across the UK from here https://data.gov.uk/dataset/road-accidents-safety-data . We used the following files:

2014 Road Safety - Accidents 2014
2014 Road Safety - Vehicles 2014
2014 Road Safety - Casualties 2014
Lookup up tables for variables

Local copies were downloaded and stored in ./GraphHackData

accidents = pd.read_csv('./GraphHackData/DfTRoadSafety_Accidents_2014.csv')
vehicles = pd.read_csv('./GraphHackData/DfTRoadSafety_Vehicles_2014.csv')
casualties = pd.read_csv('./GraphHackData/DfTRoadSafety_Casualties_2014.csv')

Some strange characters are in some of the column names so let’s strip them out and then then merge accidents and casualties on Accident_Index

accidents = accidents.rename(columns={'Accident_Index': 'Accidents_Index'})
vehicles = vehicles.rename(columns={'Accident_Index': 'Accidents_Index'})
casualties = casualties.rename(columns={'Accident_Index': 'Accidents_Index'})

accidentsDF = pd.merge(accidents, casualties, on='Accidents_Index')

accidentsDF.head()

	Accidents_Index	Location_Easting_OSGR	Location_Northing_OSGR	Longitude	Latitude	Police_Force	Accident_Severity	Number_of_Vehicles	Number_of_Casualties	Date	...	Age_of_Casualty	Age_Band_of_Casualty	Casualty_Severity	Pedestrian_Location	Pedestrian_Movement	Pedestrian_Road_Maintenance_Worker	Casualty_Type	Casualty_Home_Area_Type
0	201401BS70001	524600	179020	-0.206443	51.496345	1	3	2	1	09/01/2014	...	49	8	3	0	0	0	8	1
1	201401BS70002	525780	178290	-0.189713	51.489523	1	3	2	1	20/01/2014	...	27	6	3	0	0	0	1	-1
2	201401BS70003	526880	178430	-0.173827	51.490536	1	3	2	1	21/01/2014	...	27	6	3	0	0	0	3	1
3	201401BS70004	525580	179080	-0.192311	51.496668	1	3	1	1	15/01/2014	...	31	6	3	1	1	2	0	1
4	201401BS70006	527040	179030	-0.171308	51.495892	1	3	2	1	09/01/2014	...	32	6	3	0	0	0	9	1

5 rows × 46 columns

Which accidents are in London?

Santander bikes are only available in London so we also need to be able to filter by whether an accident is in London. Accidents are all assigned to LSOA (Lower Layer Super Output Area).

We identify all the LSOAs in London using this ref: http://data.london.gov.uk/dataset/lsoa-atlas

london = pd.read_excel('./GraphHackData/lsoa-data.xls', sheet='iadatasheet1', skiprows=2)
lsoa = set(london.Codes)

#Add a boolean column to the accidents dataframe to describe if in London
accidentsDF['in_London'] = accidentsDF.LSOA_of_Accident_Location.map(lambda x: x in lsoa)

From the DoT lookup table we identify that any casualty listed as 1 is a cyclist and hence we can now find all accidents in London in 2014 where the casualty was a cyclist

cyclingAccidents = accidentsDF[(accidentsDF['in_London'] == True) & (accidentsDF.Casualty_Type == 1)]

#Example incident
example = cyclingAccidents.loc[59,]
example

Accidents_Index 201401BS70065
Location_Easting_OSGR 526610
Location_Northing_OSGR 177280
Longitude -0.178126
Latitude 51.4803
Police_Force 1
Accident_Severity 3
Number_of_Vehicles 2
Number_of_Casualties 1
Date 08/02/2014
Day_of_Week 7
Time 18:20
Local_Authority_(District) 12
Local_Authority_(Highway) E09000020
1st_Road_Class 3
1st_Road_Number 3220
Road_Type 6
Speed_limit 30
Junction_Detail 0
Junction_Control -1
2nd_Road_Class -1
2nd_Road_Number 0
Pedestrian_Crossing-Human_Control 0
Pedestrian_Crossing-Physical_Facilities 5
Light_Conditions 4
Weather_Conditions 2
Road_Surface_Conditions 2
Special_Conditions_at_Site 0
Carriageway_Hazards 0
Urban_or_Rural_Area 1
Did_Police_Officer_Attend_Scene_of_Accident 2
LSOA_of_Accident_Location E01002840
Vehicle_Reference 2
Casualty_Reference 1
Casualty_Class 1
Sex_of_Casualty 1
Age_of_Casualty 32
Age_Band_of_Casualty 6
Casualty_Severity 3
Pedestrian_Location 0
Pedestrian_Movement 0
Car_Passenger 0
Bus_or_Coach_Passenger 0
Pedestrian_Road_Maintenance_Worker 0
Casualty_Type 1
Casualty_Home_Area_Type -1
in_London True
Name: 59, dtype: object

Let’s see what the two nearest Santander bike stations are to this accident …

query = "MATCH (b:Bike_station) WITH b, distance(point(b), point({{latitude:{0}, longitude:{1}}})) AS dist RETURN b.a_Id AS station_id, b.name AS station_name, dist ORDER BY dist LIMIT 2".format(example.Latitude, example.Longitude)
cypher = graph.cypher
result = cypher.execute(query)

result

 | station_id | station_name | dist
---+------------+---------------------------------+-------------------
1 | 746 | Lots Road, West Chelsea | 99.27002411121134
2 | 649 | World's End Place, West Chelsea | 227.456404989975

** So first piece of insight appears to be not to cycle at “World’s End”! **

Before generating the graph of the accidents we need to convert many of the numerical classifications into their human readable form to make things easier to interpret

#Conversion for some of the accident variables to huamn readable form

roadClass = {1: "Motorway",
 2: "A(M)",
 3: "A",
 4: "B",
 5: "C",
 6: "Unclassified"}

dow = {1: "Sunday",
 2: "Monday",
 3: "Tuesday",
 4: "Wednesday",
 5: "Thursday",
 6: "Friday",
 7: "Saturday"}

lightConditions = {1: "Daylight",
 4: "Darkness: lights lit",
 5: "Darkness: lights unlit",
 6: "Darkness: no lighting",
 7: "Darkness: lighting unknown",
 -1: "Data missing"}

weatherConditions = {1:"Fine no high winds",
 2:"Raining no high winds",
 3:"Snowing no high winds",
 4:"Fine + high winds",
 5:"Raining + high winds",
 6:"Snowing + high winds",
 7:"Fog or mist",
 8:"Other",
 9:"Unknown",
 -1:"Data missing"}

roadConditions = {1: "Dry",
 2: "Wet or damp",
 3: "Snow",
 4: "Frost or ice",
 5: "Flood over 3cm deep",
 6: "Oil or diesel",
 7: "Mud",
 -1: "Data missing"}

gender = {1: "Male",
 2: "Female",
 3: "Not known",
 -1: "Data missing"}

severity ={1: "Fatal",
 2: "Serious",
 3: "Slight"}

Now we are ready to generate accident nodes and map them to the two nearest bike stations ** N.B. we will not map if the nearest bike station isn’t within 2km of an accident as Santander bike stations are concentrated in the centre rather than across the whole of what is labelled London **

def genAccidentNodes(datum):
 """For a given row in the accidents dataframe construct the appropriate set of nodes and relationships"""
 accident = Node("Accident", a_Id = datum.Accidents_Index)
 accident['latitude'] = datum.Latitude
 accident['longitude'] = datum.Longitude
 accident['severity'] = severity[datum.Casualty_Severity]
 accident['severity_score'] = 4. - datum.Casualty_Severity
 accident['time'] = datum.Time
 graph.create(accident)
 date = graph.merge_one('Date', "value", datum.Date)
 date.properties['day_of_week'] = dow[datum.Day_of_Week]
 graph.push(date)
 rel_1 = Relationship(accident, "HAPPENED_ON", date)

 #Make vector of relationships to create
 relationships = []
 relationships.append(rel_1)

 weatherCon = weatherConditions.get(datum.Weather_Conditions, "Data missing")
 if weatherCon != "Data missing":
 weather = graph.merge_one('Weather', "condition", weatherCon)
 relationships.append(Relationship(accident, "WITH", weather))
 lightCon = lightConditions.get(datum.Light_Conditions, "Data missing")
 if lightCon != "Data missing":
 light = graph.merge_one('Light', "condition", lightCon)
 relationships.append(Relationship(accident, "WITH", light))
 roadSurfaceCon = roadConditions.get(datum.Road_Surface_Conditions, "Data missing")
 if roadSurfaceCon != "Data missing":
 roadSurf = graph.merge_one('Road_surface', "condition", roadSurfaceCon)
 relationships.append(Relationship(accident, "WITH", roadSurf))

 speed = graph.merge_one("Speed_limit", "value", np.int(datum.Speed_limit))
 relationships.append(Relationship(accident, "WITH", speed))

 #And find the nearest bike stations
 query = "MATCH (b:Bike_station) WITH b, distance(point(b), point({{latitude:{0}, longitude:{1}}})) AS dist RETURN b.a_Id AS station, dist ORDER BY dist LIMIT 2".format(datum.Latitude, datum.Longitude)
 result = cypher.execute(query)
 #Only do this for bike sations where the nearest station is less than 2km away
 if result.records[0].dist <= 2000.:
 for i,rec in enumerate(result.records):
 bikeStation = graph.merge_one("Bike_station", "a_Id", rec.station)
 tempRel = Relationship(accident, "CLOSE_TO", bikeStation, distance=round(rec.dist,2), proximity=i+1)
 relationships.append(tempRel)

 graph.create(*relationships)

Let’s run the function over all London cycling accidents

output = cyclingAccidents.apply(genAccidentNodes, axis=1)

Graphically exploring our new database

An accident node and associated properties including closest bike docking stations

Which bike docking stations are linked to the fatal cycling accidents?

What was the speed limit on the roads with fatal cycling accidents?

The most dangerous bike docking stations to cycle between

We can query neo4j to count the number of accidents between bike stations

query = """MATCH (b1:Bike_station)<-[:CLOSE_TO]-(a:Accident)-[:CLOSE_TO]->(b2:Bike_station)
WITH b1, b2, COLLECT(DISTINCT a.a_Id) AS accidents
WHERE b1.a_Id < b2.a_Id
RETURN b1.name AS station1, b1.longitude AS lon1, b1.latitude AS lat1,
 b2.name AS station2, b2.longitude AS lon2, b2.latitude AS lat2,
 size(accidents) AS num_accidents
ORDER BY num_accidents DESC;"""
result = cypher.execute(query)

The top 10 most dangerous bike station pairs are …

df = pd.DataFrame(result.records, columns=result.columns)
df.head(10)

	station1	lon1	lat1	station2	lon2	lat2	num_accidents
0	Clarence Walk, Stockwell	-0.126994	51.470733	Binfield Road, Stockwell	-0.122832	51.472510	61
1	Shoreditch Court, Haggerston	-0.070329	51.539084	Haggerston Road, Haggerston	-0.074285	51.539329	48
2	Islington Green, Angel	-0.102758	51.536384	Charlotte Terrace, Angel	-0.112721	51.536392	32
3	Ravenscourt Park Station, Hammersmith	-0.236770	51.494224	Hammersmith Town Hall, Hammersmith	-0.234094	51.492637	32
4	Bricklayers Arms, Borough	-0.085814	51.495061	Rodney Road , Walworth	-0.090221	51.491485	30
5	Napier Avenue, Millwall	-0.021582	51.487679	Spindrift Avenue, Millwall	-0.018716	51.491090	28
6	Ada Street, Hackney Central	-0.060292	51.535717	Victoria Park Road, Hackney Central	-0.054162	51.536425	26
7	Wandsworth Rd, Isley Court, Wandsworth Road	-0.141813	51.469260	Heath Road, Battersea	-0.146545	51.468669	24
8	Caldwell Street, Stockwell	-0.116493	51.477839	Binfield Road, Stockwell	-0.122832	51.472510	23
9	Stebondale Street, Cubitt Town	-0.009205	51.489096	Saunders Ness Road, Cubitt Town	-0.009001	51.487129	23

Plotting everything out looks like this

df['mean_longitude'] = (df.lon1+df.lon2)/2.
df['mean_latitude'] = (df.lat1+df.lat2)/2.
df2 = df[df.num_accidents > 10]

%matplotlib inline
import geopandas as gpd
import matplotlib.pyplot as plt
import mplleaflet

plt.rcParams['figure.figsize'] = 14, 10
fig = plt.figure()
plt.plot(cyclingAccidents.Longitude, cyclingAccidents.Latitude, 'b.', alpha=0.4, label='cycling accidents (2014)')
plt.plot(stationsDF.long, stationsDF.lat, 'rs', alpha=0.5, label='docking stations')
plt.scatter(df2.mean_longitude, df2.mean_latitude, s=df2.num_accidents*25, c='yellow', alpha=0.5, label='Most accidents')
plt.xlim(-0.25, 0.)
plt.ylim(51.45, 51.55)
#plt.legend(loc='lower right')

(51.45, 51.55)

Can combine this with a map to get a better feel for where in London we are

mplleaflet.display(fig=fig, tiles='osm')

Conclusions

Some interesting first results but more work needed

It’s great to see that the initial analysis worked and using Neo4j made our analysis easier and there is a lot more data stored in there that could be analysed at a later date. What at first appears surprising is that the “danger stations” that we have identified appear to gnerally bound the region that Santander bikes are available in, however, we cannot confirm these are correct without correcting for a couple of additional observational biases:

Normalise for the amount of journeys starting/ending at each station, i.e. do more accidents happen because more people are riding in these parts of London.
The density of bike docking stations is not uniform so those on the periphery may be being assigned more accidents based upon fewer stations to assign the accidents to.

DataDiving into corporate ownership

Sun, 01 Jan 2017 00:00:00 +0000

On the 12-13 November 2016 DataKind UK hosted a DataDive examining corporate ownership data. The event was run in partnership with Global Witness, a not-for-profit organisation that campaigns against environmental and human rights abuses that are often derived from the exploitation of natural resources and corruption in the global political and economic system; see the great video here for a better explanation. They carry-out investigations that expose these abuses and came to the DataDive in the hope that the data science volunteers could help them explore the the world’s first open data register of “beneficial owners” or “people with significant control” of companies registered in the UK.

Using date from OpenCorporates the goal of the DataDive was to explore the new “beneficial owners” data to see what it reveals about potential cases of tax evasion and corruption. To this end the ~30 or so volunteer data scientists split into 3 teams:

Firstly, an analytics team to derive initial insights into the “beneficial ownership” and what it could say about companies within the UK.
Combining this data with other data sets could the register yield new investigative leads.
Could we map out the ownership network in such a way that we could query the data in new ways.

I chose to work with Team 3, and ultimately we decided to try and build a graph database out of the data we had at hand. So before I spend the rest of this blog post explaining what we figured out, let me say that the other two teams came up with some fantastic results and taking a look at the DataKind blog post and Global Witness blog post is well worth your time. For starters it was found that 3,000 companies had listed their beneficial owner was a company with a tax haven address in contravention of the rules on declaring beneficial owners.

I do need to give a shout out to Juan and Ned, two friends also at the DataDive on Team 3, who spent the entire weekend trying to figure out how to create unique identifiers for people while also making sure to match the same person e.g. Mr Alfred Hitchcock is the same as Sir Alfred J. Hitchcock. Definitely not an easy task but they made valiant progress

Diving into building a new Neo4j database in <24 hours

Immediately upon looking at the data we had in hand it was clear that it could be well represented as a network and hence we thought that graph analytics might offer some new innovative approaches to exploring the data. I was one member of our team who took on the challenges of converting our sketch of behaviour into a useable graph database that could be easily queried and explored.

The fact that I and a friend Stew, who was also at the datadive had some experience with Neo4j and that there is a free community edition for experimenting we chose that as the tech basis for what we would build. As illustrated in the sketch above we would construct nodes of: people (owners); companies; countries; sectors (we didn’t complete this part). The relationships would then connect these nodes together through ownership and location.

The goal ultimately was to build a system that enabled easy querying to find links within corporate structures. So rather than focus on technical details of how we ultimately built it all I thought I’d share some of the sorts of queries we ended up running and the patterns they revealed.

Visualising patterns of corporate ownership

The following are visualisations that have come directly from running CYPHER queries on the neo4j database that we built. For the case of these renders I’ve anonymised the individual people and companies as the purpose here is to demonstrate the sorts of structures and patterns that can be extracted. The colour of the nodes indicate their type:

Companies are blue
People are green
Countries are red
Postal codes are purple

These nodes are then joined via a variety of relationships including:

CONTROLS - indicates a controlling relationship between a person or company and another company
REGISTERED_IN - indicates the country or postal code where something is registered
CITIZEN_OF - indicates citenship of people

Self-control: One of the goals of the “beneficial ownership” requirement is that you see which people and corporations ultimately own a company, so it was intriguing to find that a number of companies reporting that they were controlled by themselves. In all likelihood this is actually confusion in filing the new ownership information rather than actual attempts at obfuscating ownership.