So I built Statica: a wrapper for the AppSec tools I trust.

Some useful AppSec tools

Statica runs a hand-picked set of tools to uncover security issues, inspired by classics like metric-fu:

• Trivy: Finds secrets in code.
• Semgrep: Flags bugs and security holes.
• Bearer: Sniffs out secrets and risky code.
• Lizard: Spots complexity in code.
• RetireJS: Catches vulnerable JavaScript libraries.
• CPD: Finds copy-paste code.
• Checkov: Scans IaaC (e.g. Terraform) code for security slip-ups.

How Statica works

Statica is a CLI tool. Point it at your codebase, and it runs each tool in sequence, then rolls up the results into a single HTML or console report. Easy install and updates via Homebrew on macOS and Linux.

Statica saves tool outputs in SARIF format and merges them for a single view of a codebase’s weak spots.

Example

brew install simpsonjulian/statica-tap/statica
statica /path/to/codebase html
open /path/to/codebase/statica.html

What’s next?

More checks, and better insights into the results.

Conclusion

Statica is for anyone who wants to get SAST running on their codebase, fast. Try it and let me know how it goes.

I’ve needed to make a YAML pipeline on Azure DevOps, using ADO Git hosting. It’s not the edgiest stack out there, but as a consultant it’s rare that you get to choose the tools you use. Sometimes you just need to get something automated today. It’s easy to set up pipeline triggers from changes in the Git repository:

trigger:
  branches:
    include:
      - '*'

Later on, one wants to ensure that a job runs in Pull Requests, so that people can get feedback on those PR’s before they get merged: it’s a powerful way to go from “LGTM” to”let’s fix that vulnerability first”.

pr:
  branches:
    include:
      - main
      - feature-*

It’s possible to spend a lot of time wondering why this doesn’t work. Here’s the thing: presently, it doesn’t work for Azure’s Git hosting. You’ll need to go to the Repository in Azure DevOps and set up a Build validation policy. This is a pointy-click change, but you can use the Azure CLI if you want a hands-off workaround. I can empathise with the Azure DevOps team here: they’ve had to evolve this product over 17 years to keep up with changing fashions in development. This feature hasn’t been migrated to YAML yet. The world has changed a lot, Enterprise customers are looking for stability, and there’s a workaround to keep the pipelines flowing.

We want to stop bad things from happening early, not pick them up after publishing or deployment.

Insecure containers expose the system to application vulnerabilities. Let’s go find some!

Linting

You want to know if you’re building containers with weak security. How do you find out? By Linting the Dockerfile before you build it. I chose Hadolint as the tool to do this on a sample application, and added it to the Makefile:

lint:
	docker run --rm -i hadolint/hadolint < Dockerfile

docker: app.war lint
	docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-$(VERSION).war -t demo:$(VERSION) .

Hadolint found issues with my 6 line Dockerfile, some of which could have impacted security: for example you shouldn’t leave package cache files sitting around if you happened to install a package in your Docker build.

Scanning

Any good container repository should let you scan for vulnerabilities: Azure and AWS Container repositories do, the Docker service also does. But why wait? I added Trivy (hat tip to Phil) in order to expose any issues. Oh my Lord there were issues:

demo:1.0.0 (alpine 3.14.0)

Total: 36 (UNKNOWN: 0, LOW: 0, MEDIUM: 6, HIGH: 26, CRITICAL: 4)

┌──────────────┬────────────────┬──────────┬───────────────────┬───────────────┬─────────────────────────────────────────────────────────────┐
│   Library    │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                            Title                            │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ apk-tools    │ CVE-2021-36159 │ CRITICAL │ 2.12.5-r1         │ 2.12.6-r0     │ libfetch before 2021-07-26, as used in apk-tools, xbps, and │
│              │                │          │                   │               │ other products, mishandles...                               │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-36159                  │
├──────────────┼────────────────┼──────────┼───────────────────┼───────────────┼─────────────────────────────────────────────────────────────┤
│ busybox      │ CVE-2021-42378 │ HIGH     │ 1.33.1-r2         │ 1.33.1-r6     │ busybox: use-after-free in awk applet leads to denial of    │
│              │                │          │                   │               │ service and possibly...                                     │
│              │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2021-42378                  │
│              ├────────────────┤          │                   │               ├─────────────────────────────────────────────────────────────┤
│              │ CVE-2021-42379 │          │                   │               │ busybox: use-after-free in awk applet leads to denial of    │

There were so many issues found in the Docker image and the Java app inside it, that I had to pause writing and go fix it. Even then I couldn’t find a clean Docker image, and there are still transitive dependencies from Spring Boot that cause noise. That’s a challenge managing vulnerabilities in a live project, so I configured it to exclude unfixed findings, and only fail at a threshold. Right now it fails the build if there are Critical vulnerabilities discovered.

docker: app.war lint
	docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-$(VERSION).war -t demo:$(VERSION) .
	trivy image demo:${VERSION} --exit-code 1  --ignore-unfixed  --severity CRITICAL #,HIGH

It takes about 2 seconds to build and check the Docker image (excluding building the toy Spring Boot app). See below for details. Doing this for all your container images gives a high ROI for your time: you’ll reduce your exposure to application and container vulnerabilities.

time make docker
docker run --rm -i hadolint/hadolint < Dockerfile
docker build --platform linux/amd64 --build-arg JAR_FILE=build/libs/\demo-1.0.0.war -t demo:1.0.0 .
[+] Building 1.0s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                     0.0s
 => => transferring dockerfile: 37B                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/openjdk:18                                                                                                                            0.9s
 => [internal] load build context                                                                                                                                                        0.0s
 => => transferring context: 104B                                                                                                                                                        0.0s
 => [1/2] FROM docker.io/library/openjdk:18@sha256:ac8bc800c76446207675f93939a8977e032043a182c1a6be23c6cdd300db0cc5                                                                      0.0s
 => CACHED [2/2] COPY build/libs/demo-1.0.0.war /app.war                                                                                                                                 0.0s
 => exporting to image                                                                                                                                                                   0.0s
 => => exporting layers                                                                                                                                                                  0.0s
 => => writing image sha256:8ff141dc971724c6602aa51bb709b6ae804c720cdfa44e19fc866a4ffe214a8a                                                                                             0.0s
 => => naming to docker.io/library/demo:1.0.0                                                                                                                                            0.0s
trivy image demo:1.0.0 --exit-code 1  --ignore-unfixed  --severity CRITICAL #,HIGH
2022-10-24T12:45:49.662+1300	INFO	Vulnerability scanning is enabled
2022-10-24T12:45:49.662+1300	INFO	Secret scanning is enabled
2022-10-24T12:45:49.662+1300	INFO	If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-10-24T12:45:49.662+1300	INFO	Please see also https://aquasecurity.github.io/trivy/v0.32/docs/secret/scanning/#recommendation for faster secret detection
2022-10-24T12:45:49.692+1300	INFO	Detected OS: oracle
2022-10-24T12:45:49.692+1300	INFO	Detecting Oracle Linux vulnerabilities...
2022-10-24T12:45:49.699+1300	INFO	Number of language-specific files: 1
2022-10-24T12:45:49.699+1300	INFO	Detecting jar vulnerabilities...

demo:1.0.0 (oracle 8.6)

Total: 0 (CRITICAL: 0)


real	0m2.011s
user	0m0.439s
sys	0m0.095s

So I downloaded a year’s worth of withdrawals in CSV format, and did this:

egrep -i "powerco|gasco" account.csv | awk -F "," '{s+=$4} END {printf "%.0f\n", s}'

I could write the whole thing in awk, but these things evolve by tacking commands onto the pipelines, and there’s no need to get clever.

We spent more than I’d expected to, so my next job is to find out what is chewing up so much energy. Of course, normal people would use a spreadsheet.

I started from first principles with a basic Spring Boot application, tried some different distributions of Kubernetes, and made an application deploy. Then made the thing visible from my local computer. That took a long time. Here’s what I learned:

Apple Silicon has teething issues: I love my M1 MacBook Pro. I found the following gumption traps in my first session:

• The official Kubernetes tutorial doesn’t work by default on ARM, as it refers to images that aren’t binary compatible.
• Docker will throw occasional errors - either crashing on start, or having networking glitches. I’m sure that will get fixed. I’m using Rancher in the interim.
• Minikube didn’t work out of the box. Many posts online suggest configuring minikube to use Docker, which didn’t work for me.
• You’ll get used to setting the platform parameter everywhere, to run Intel/AMD containers instead of ARM.

The ecosystem is a loud bazaar: On the desktop there’s several different distributions of Kubernetes. Each cloud provider supports at least one implementation. The ecosystem that interacts with these is huge. Similar itches are being scratched many different ways. My view is that you needs to work out what Dev and Ops cultures matter in your team:

• If GitOps is your thing, then you might like ArgoCD
• If you’re into configuration as code, you might like CDK8S
• If you like templating YAML or JSON, there are many different tools (Kustomize, Tanka out there for you.

It’s hard to choose tools correctly the first time: as communities and systems evolve, yesterday’s right choice might be today’s technical debt. If you can keep an open mind about where to apply different approaches, then you may be able to incubate newer tools and get some payoff from them. If you stick to the tools that you’re comfortable with, you may find yourself with an obsolete stack.

Where I got to with my playpen application: I’m building local images using the Docker CLI against Rancher, orchestrated with Make, and calling down to Gradle to build Java stuff. After experimenting with plain YAML and kubetcl, exploring Tanka, I finished with the Terraform provider for Kubernetes. I would want to deploy AKS or EKS infrastructure with Terraform, and to kick the tyres I will continue with Terraform to get apps lifted and shifted to Kubernetes. That won’t scale forever, and at some point I’ll find the right tools to manage the inside of the cluster. But for today, I’m going to add one new tool to the toolbelt.

Update: I got around to Securikube Part 1.

When you’re getting a pen test for your web application, testers are duty bound to give you findings about your missing HTTP headers. It’s important, but it also muddies the waters with findings that anyone could generate. If you visit Security Headers and ensure that your application has a clean scan before the pen testers show up, you’ll get a cleaner report.

I scanned this site last week and got a D. After reading the recommendations and adding some Zeit config, it’s an A+. Probably not a huge acheivement for this static site, but a good tool to have in the shed.

I’ve since pulled nearly everything back to AWS so I can manage it all with Terraform in a GitHub repo. That raises a question: how secure are my AWS resources? Inspecting stuff with AWS is possible, with tools like AWS Security Hub. It would be nice to know that I were making secure Terraform code before I pushed it and applied. I chose Checkov. There are other tools out there, like Snyk IaC.

I installed the Checkov package from PiPi using Pipenv, and then invoked pipenv shell to call the checkov script from a Makefile. Here’s the kind of output that Checkov gives you:

(terraform) bash-3.2$ checkov --quiet -d .; exit

       _               _
   ___| |__   ___  ___| | _______   __
  / __| '_ \ / _ \/ __| |/ / _ \ \ / /
 | (__| | | |  __/ (__|   < (_) \ V /
  \___|_| |_|\___|\___|_|\_\___/ \_/

By bridgecrew.io | version: 2.0.338

terraform scan results:

Passed checks: 91, Failed checks: 3, Skipped checks: 43

Check: CKV_AWS_19: "Ensure all data stored in the S3 bucket is securely encrypted at rest"
	FAILED for resource: aws_s3_bucket.media-build-doctor-com
	File: /build-doctor.com.tf:84-90
	Guide: https://docs.bridgecrew.io/docs/s3_14-data-encrypted-at-rest

		84 | resource "aws_s3_bucket" "media-build-doctor-com" {
		88 |   bucket = "foo.build-doctor.com"
		89 |   acl = "private"
		90 | }

Here’s what I learned:

Refactor for a better time

Checkov will tell you in no uncertain terms that your Terraform code is insecure; you’ll end up questioning your life choices. It was especially bad for my poorly composed Terraform code - I had lot of similar resources in my code, each of which picked up many warnings.

Doing the right thing and extracting modules for my static websites was a good thing - it drove consistency and reduced the howls of outage from Checkov.

Apply and test often

The age old struggle between convenience and security played out again - I spent a couple of hours happily smashing security vulnerabilities that Checkov reported, before finding that I’d completely broken the sites, with no access from S3 buckets to Cloudfront. The log buckets were so tightly locked down that I couldn’t even see what happened. After quite some time kicking tyres, I rolled the changes back.

I made some some really simple tests using curl, that I can easily run from make. This gives me confidence that the websites still work. I try to get early feedback by applying the changes in Terraform and then testing afterward.

Suppress the false positives

The many findings about S3 security make a great deal of sense in many contexts - the way we use S3 has changed during the lifetime of the S3 service. Nobody wants to win a Bucket Negligence Award. In my case, where I host some very basic static websites, I’m happy to suppress those:

resource "aws_s3_bucket" "bucket" {
  #checkov:skip=CKV2_AWS_6:We have a public access block
  #checkov:skip=CKV_AWS_18:this kind of content doesn't need encryption
  #checkov:skip=CKV_AWS_19:this kind of content doesn't need encryption
  #checkov:skip=CKV_AWS_20=Public ACLs are fine for this
  #checkov:skip=CKV_AWS_52:MFA delete hard to apply via TF
  #checkov:skip=CKV_AWS_144:Cross Region Replication not important
  #checkov:skip=CKV_AWS_145:We're not going to encrypt public website stuff

Suppression helps me to focus on the things that matter in my context.

I’ve collected a few easy options for deploying SAST tools in this post.

First on the list is SonarQube. As it supports many tools under the hood, it can give you an overwhelming list of things to fix. With a bit of setup, it’s useful for tracking the health of a codebase over time. The neat thing is that it will also detect and track security issues. If you’re being asked by clients about your SAST or DAST tooling, it’s a great way to check the box. SonarQube also works with the SonarLint IDE plugin, which is very helpful. You’ll need to buy the cloud service, or run your own SonarQube server.

Next up, is GitHub Advanced Security. Being a GitHub feature, it’s tightly integrated with GitHub and can be configured to run on push, or as an Action. You can also add third party scanners, and use webhooks to add Jira tickets to the backlog. Advanced Security needs an Enterprise plan, so if you’re looking to throw money at the problem, this could be the solution.

Finally, there’s two other tools that I think deserve a mention: Snyk, because they have good IDE support (why not squash them at the source?), and it integrates with BitBucket and GitLab. Also, of note is Muse; Sonatype acquired this recently and will be releasing hosted and free versions. It’ll be branded as Sonatype Lift and will integrate with the Maven ecosystem, to help address supply chain compromises.

• The site still had stylesheets loading for Bulma, which I stopped using in favour of my own minimal CSS. I also had references to Bulma classes because the stylesheet was still there.
• I hadn’t set the JEKYLL_ENV environment variable in Vercel, which stopped Google Analytics from loading, but I still had leftover Google Analytics code lying around, and Discuss includes rotting away.
• There was a bad client side contact form made for for another site in 2017 or sot. It’s replaced with a third party service, but I’m tempted to write something else to accept forms from this site.
• The one image that loads on the site was slowing down the first render, so I fired up ImageMagick and changed it to a more performant format.
• I also added compression to make the site load even faster.

Very satisfying to fix tech debt. There’s always tech debt.

Staying alive beats everything else. Like a living thing, startups will endure all sorts of horrors to stay alive. Crappy work conditions, penny-pinching, super economy travel: all necessary unless you have the enough runway to upgrade. It’s the job of the management team to keep the business afloat, so don’t complain about having to buy your own lunch. The present amount of funding available means that startup employees get amazing benefits: it’s not normal.

Silos will hurt your startup. It’s easier than ever for organisations to spread around the world. It’s also easy to create geographic as well as organisational silos. You need to work out how to prevent that in your organisation: having Sales based in one region and Engineering in another can easily lead to silos. We’re tribal creatures, and will naturally create ingroups and outgroups based on department and geography. Plan accordingly.

Risk and compliance will matter. Sure, go ahead and pivot all you like while you’re trying to find your startup’s place in the world. Kick as many cans down the road as you can: you might not even stay on that road. Once you’re on that road: start investing in the tools to make compliance and security easier. You don’t have to start with much; perhaps some advice on how to get started. I’ve seen several organisations make compliance difficult, or be unaware of the risk that they’re running by uh, not managing risk.

While you’re reading about risk: after ten long years in startups I’m moving back to consulting. I’ll be joining Safe Advisory at the beginning of May to do security and DevSecOps advisory. I’m looking forward to the change: helping people to demystify security and fearlessly secure their organisations feels like a worthwhile pursuit. It also gets me out of the house from time to time.