Methodology · A1131

DEBUGGING RULES
FOR EFFICIENCY

Rulest Output 4.3M Rules ~70M Hashes Pareto Analysis

Since rules extracted with rulest can be used directly in tools like hashcat, after generating a substantial number of them I decided to briefly document the process of building highly optimal hashcat rules from that data in this short guide.

12 rulest runs
depths 1–10
4.38M unique rules
after dedup
~70M MD5 debug hashes
starting corpus
~4h RTX 3060 Ti
63 MB wordlist
Step-by-Step Workflow
01

Collect rules from multiple rulest runs

From 12 runs of the tool I obtained 12 rule files covering depths 1–10 (up to 10 operator+argument pairs per rule line).

bash
$ tree
.
├── found_chains.phase0.txt
├── found_chains.txt
├── rulest_output.phase0.txt
├── rulest_output.txt
├── rulest_raw.rule
├── rulest_rules_ga.rule
├── rulest_rules_no_ga.rule
├── rulest_rules_skull.rule
├── rulest_rules_strip_1p7m.rule
├── rulest_rules_strip_500k.rule
├── rulest_rules_strips.phase0.txt
└── rulest_strip_hashmob.rule
0 directories, 12 files
02

Merge and sort by frequency

After sorting by occurrence (frequency) I ended up with a file containing 4,384,527 unique rules. Not all of them are necessarily effective, so to find the most valuable ones I used the tried-and-true method of debugging rules against a large set of 32-hex hashes. The original plan was to use 160 million hashes, but an SSH connection drop mid-session made it impossible to repeat the full process — the debug run was ultimately completed against approximately ~70 million hashes, which still provides a solid statistical basis for frequency analysis.

bash
$ cat * | uniq -c | sort -nr | cut -c 9- | awk '!seen[$0]++' > rulest2debug.rule

$ wc -l rulest2debug.rule
4384527 rulest2debug.rule
03

Run the debug pass with hashcat

I used the following script to run the debug pass:
→ https://github.com/A113L/bucket — run_debug.sh

Hardware reference: On an RTX 3060 Ti with a 63 MB wordlist this took roughly 4 hours using hashcat flags --debug-mode=1 and --debug-file=rulest_debug_data.txt.
hashcat
$ hashcat -a 0 -m 0 MD5_debug_hashes.txt wordlist.txt \
    -r rulest2debug.rule \
    --debug-mode=1 \
    --debug-file=rulest_debug_data.txt
04

Process debug output through concentrator

The resulting debug file was then processed through concentrator, which includes a Pareto-curve analysis mode designed for large sets of repeating debug data. Rules that appear most frequently across the debug output are the most valuable — that is the core principle behind the selection.

To use concentrator on the extracted rule set, simply run it without any arguments — this launches the program in interactive mode. The script begins by counting the total number of rules, so you know upfront how many you are working with.

concentrator
$ ./concentrator

Input Configuration:
Enter rule files/directories (space-separated): rulest_debug_data.txt

Analysing Input Data...
✅ SUCCESS: Rule file: rulest_debug_data.txt
Quick Analysis:
  Files:          1
  Sampled rules:  24,299,147
  Est. total:     24,299,147
  Unique sample:  1,048,061
  Max rule len:   39

Recommendation: Low uniqueness → Extraction
  Long rules detected → consider functional minimization later.

Processing Mode:
  1 – Extract top existing rules
  2 – Generate combinatorial rules
  3 – Generate Markov rules
  Recommended: Mode 1
Select mode (1-3): 1
Top rules to extract [10000]: 24299147
Use statistical sort? [y/N]: n

Global Settings:
Output base name ['concentrator_output']: 
Max rule length to process [31]: 
Enable GPU acceleration? [Y/n]: y
Process entirely in RAM? [y/N]: n

Output Format:
  1 – Standard line
  2 – Expanded (space-separated operators)
Select (1-2): 2
Temp directory [system default]: .

Configuration Summary:
  Mode:          extraction
  Input paths:   1 location(s)
  Output base:   concentrator_output
  Max rule len:  31
  GPU:           Enabled
  In-memory:     No
  Output format: expanded
  Top rules:     24,299,147
  Stat sort:     No

Start processing? [Y/n]: y
Active Mode: EXTRACTION
Output File: concentrator_output_extracted.rule
Output Format: expanded
✅ SUCCESS: OpenCL initialised on: NVIDIA GeForce RTX 3060 Ti
✅ SUCCESS: GPU Acceleration: ENABLED

 Collecting Rule Files (recursive, max depth 3) 
✅ SUCCESS: Rule file: rulest_debug_data.txt
✅ SUCCESS: Found 1 rule files.
ℹ️  INFO: Using temporary directory: .

 Parallel Rule File Analysis 
ℹ️  INFO: Parallel analysis of 1 files using 1 processes...

Enter enhanced interactive mode? (Y/n): y

================================================================================
                  ENHANCED RULE PROCESSING – INTERACTIVE MENU                   
================================================================================
Initial dataset: 887,270 unique rules

--------------------------------------------------------------------------------
ADVANCED FILTERING OPTIONS:
 (1) Filter by MINIMUM OCCURRENCE
 (2) Filter by MAXIMUM NUMBER OF RULES (top N)
 (3) Filter by FUNCTIONAL REDUNDANCY [RAM intensive]
 (4) INVERSE MODE – keep rules BELOW the cut-off rank
 (5) HASHCAT CLEANUP – validate (CPU/GPU modes)
 (6) LEVENSHTEIN FILTER – remove similar rules
 (7) TOGGLE OUTPUT FORMAT (currently: expanded)

ANALYSIS & UTILITIES:
 (p) PARETO analysis
 (s) SAVE current rules
 (r) RESET to original dataset
 (i) Dataset information
 (q) QUIT
--------------------------------------------------------------------------------
Enter choice: p
Large file tip: If the file is large, do not load everything into RAM. The script will instead use temporary files to sort the dataset by frequency. This takes longer but completes the task regardless of file size. You can also opt in to GPU-accelerated rule validation at this stage.
Pareto Curve Analysis (80/20)

Once sorting is complete you will be dropped into a submenu where you can find the Pareto curve analysis along with milestone breakpoints showing exactly how many rules are needed to cover a given percentage of the entire initial dataset.

Principle: Rules that repeat most frequently across the debug output carry the highest real-world value. The top 1–5% of rules by frequency typically covers the vast majority of recoverable hashes.
Milestone Coverage Breakpoints

Pareto milestones from concentrator — rules needed to reach each coverage percentage of the full 887K-rule corpus (21.4M total occurrences).

10% coverage
42 rules  (0.0%)
20% coverage
200 rules  (0.0%)
30% coverage
574 rules  (0.1%)
40% coverage
1,447 rules  (0.2%)
50% coverage
3,678 rules  (0.4%)
60% coverage
8,803 rules  (1.0%)
70% coverage
21,505 rules  (2.4%)
80% coverage
48,834 rules  (5.5%)
90% coverage
138,936 rules  (15.7%)
95% coverage
283,945 rules  (32.0%)
99% coverage
672,981 rules  (75.8%)

This makes it straightforward to cut the ruleset down to a practical size — for most cracking jobs the top 1–5% of rules by frequency covers the vast majority of real-world hits.

Output Rule Files

Using the Pareto milestones as cut points, four rule files were produced — one for each practical use case, from a large comprehensive set down to a tiny high-precision set:

bash
$ wc -l *.rule
 500001 rulest_large.rule
 150001 rulest_medium.rule
  25001 rulest_small.rule
    251 rulest_tiny.rule
 675254 total
Benchmark Comparison

Results from the rules comparison spreadsheet. Each of the four output files is compared against the best-in-class rule set in its weight category, and against other existing rulest_* variants.

rulest_large.rule · 500k
Recovered: 44.54%
Speed: 2,814 H/h
Rules: 499,996
Best-in-class at 500k: Fordyv3-500k 48.92%
collection500k: 43.38% · hashcracky_500k: 38.87%
→ Competitive — beats collection500k and hashcracky by a solid margin.
Note: overall leaderboard now led by Fordyv4a (56.94%, 4M rules) and A11313M (53.97%, 3.2M rules).
rulest_medium.rule · 150k
Recovered: 38.63%
Speed: 7,943 H/h
Rules: 149,996
Best-in-class at 150k: HashMob.150k 44.74%
0xV-Sappuka.skull.150k: 39.60%
→ Narrowly below the skull optimized variant, decent for an unoptimized first pass.
rulest_small.rule · 25k
Recovered: 26.23%
Speed: 28,780 H/h
Rules: 24,996
Best-in-class at ~25k: concentrator_MT_25000 31.82%
rockyou-30000: 26.21%
→ Virtually tied with rockyou-30k, well behind the concentrator MT variant — room for improvement with further minimization.
rulest_tiny.rule · 246
Recovered: 4.34%
Speed: 4,838 H/h
Rules: 246
Best-in-class at ~250: concentrator_MT_250 10.57%
cyclone_250: 9.33% · hashpwn_250: 8.80% · best64: 4.80%
→ Below average for the category — tiny sets are extremely sensitive to which rules land in the cut.
Comparison with other rulest_* variants

The existing rulest_* family in the benchmark includes rulest_rules_strip_1p7m (44.65%, 1.7M rules), rulest_rules_strip_1p1m (43.98%, 1.1M rules), rulest_rules_strip (38.10%, 332k rules), rulest_rules_ga (31.76%, 643k rules), and rulest_rules_no_ga (31.35%, 329k rules). The new rulest_large at 500k sits right in the middle of that family — better recovery than the strip and GA variants while being a fraction of the size of the 1M+ strip files. A good trade-off for most use cases.

Overall leaderboard context

The benchmark leaderboard has expanded significantly. New top entries include Fordyv4a (56.94%, 4M rules), Fordyv4b (56.28%, 6.9M rules), A11313M (53.97%, 3.2M rules), sapphire_v1 (52.06%, 1.27M rules), and CakeV1 (52.04%, 1M rules). These large rule sets push the ceiling considerably above the previous 48–49% range. The rulest_* output files remain competitive within their respective size categories, but the top of the overall leaderboard is now materially higher.

Key takeaway from Pareto: Just 42 rules cover 10% of cracking value. 3,678 rules get you to 50%. You don't need half a million rules to do meaningful work — the concentration of value at the top is extreme. rulest_small (25k rules, ~70% coverage milestone) gives a genuinely solid crack rate at blazing speed for a single GPU session.
Good Rules on a Budget GPU

One thing worth highlighting: rulest runs comfortably on cards with less than 4 GB of VRAM. The entire ruleflow — rule generation, debug pass, concentrator analysis — can be run on modest hardware most people already own. You don't need a flagship GPU to produce a competitive rule set. The debug pass itself is the most time-consuming part, but even that is a one-time cost. The resulting files can then be reused across any number of jobs at no additional overhead.

Combined with the Pareto insight that a few thousand high-frequency rules cover a disproportionate share of real-world hashes, this means the barrier to having solid, personalized rule sets is much lower than it appears.


Update & Respin
Round 1 · Re-Debugging Rulesets r1

After a short break — and fixing a few bugs in rulest related to generation and validation — I decided to run a quick second round (tagged r1) using the rules already acquired, this time debugging against the hashmob.tiny.found dictionary (~2 MB) rather than a full-size corpus. The rule pool itself was also refined and capped at depth 1–8 instead of the previous 1–10.

Trimming the maximum chain length is a deliberate compromise, not a step back. Longer rules are disproportionately expensive to debug and score worse against a large hash database once the operator count climbs, so they yield fewer matches per unit of debug time. Capping at 8 keeps the rule space dense where it's actually worthwhile — this was somewhat confirmed by getting 6.75M rules out of seven rulest runs using comparatively small input files. That first approach was roughly hashmob.mini vs. rockyou in scale. For comparison, rulest on hashmob.micro vs. hashmob.small alone extracts 3.2M+ deduplicated rules in balanced RCR mode at the default depth of 6 — plenty to justify a debug pass as a first filtering step.

Time budget: a tight 3 hours of debugging followed by 3 hours of extraction (RCR, rulest, concentrator + Pareto pass) — the numbers below are the result of that window.
6.75M unique rules
in r1 pool
~2 MB debug dictionary
hashmob.tiny.found
r1 tag suffix
for new rulesets
1–8 max rule depth
vs. 1–10 previously
bash
$ wc -l rulest2debug_r1.rule
6748072 rulest2debug_r1.rule

$ ls -lh hashmob.tiny.found
-rw-r--r-- 1 user user 2.0M hashmob.tiny.found
r1 Pareto Analysis (concentrator output)

Output of the concentrator Pareto pass on the r1 debug run — 1,256,836 unique rules with 35,020,748 total occurrences. Unsurprisingly, the top ten are all classic insecure number-append patterns.

#1$1$226,852  (0.1%)
#2$1$126,130  (0.2%)
#3$0$125,720  (0.2%)
#4$6$925,340  (0.3%)
#5$2$225,147  (0.4%)
#6$1$2$324,528  (0.4%)
#7$1$324,265  (0.5%)
#8$8$823,539  (0.6%)
#9$0$922,908  (0.6%)
#10$1320sz@E22,825  (0.7%)
Milestone Coverage Breakpoints — r1
10% coverage
579 rules  (0.0%)
20% coverage
3,992 rules  (0.3%)
30% coverage
12,052 rules  (1.0%)
40% coverage
25,549 rules  (2.0%)
50% coverage
45,184 rules  (3.6%)
60% coverage
73,167 rules  (5.8%)
70% coverage
113,287 rules  (9.0%)
80% coverage
178,368 rules  (14.2%)
90% coverage
328,294 rules  (26.1%)
95% coverage
513,640 rules  (40.9%)
99% coverage
919,200 rules  (73.1%)

Same shape as before, just a bigger pool underneath: 579 rules already reach 10% of total occurrences, and half the value sits in under 46k rules out of 1.26M total — even denser at the head than the previous r0 pass.

New r1 Ruleset Skeleton

The four r1 rule files are cut straight off the milestone breakpoints above — large ≈ 95%, medium ≈ 80%, small ≈ 40%, tiny ≈ 10% of cumulative coverage.

bash
$ wc -l *_r1.rule
 513641 rulest_large_r1.rule
 179166 rulest_medium_r1.rule
  25730 rulest_small_r1.rule
    585 rulest_tiny_r1.rule
 719122 total
Benchmark Comparison — r1

Real numbers from the rules comparison spreadsheet, run against all four r1 rule files.

rulest_large_r1.rule · 513.6k
Recovered: 44.52%
Speed: 2,930/h
Efficiency: 0.086675
vs. own lineage: on par with rulest_large.rule (44.54% @ 499,996) — same recovery, ~3% bigger.
vs. similar size: sits right between concentrator_MT_250000.rule (44.59% @ 250,000, half the size) and Fordyv2.rule (44.39% @ 382,629) — the extra rules past ~250k barely move the needle anymore.
rulest_medium_r1.rule · 179.2k
Recovered: 39.79%
Speed: 7,726/h
Efficiency: 0.2221
vs. own lineage: up from rulest_medium.rule's 38.63% @ 149,996 — +1.16pp recovered.
vs. similar size: ties Shooter3k_8_21_2021Cleaned.rule exactly at 39.79%, despite that file carrying 205,745 rules — ~13% fewer rules for the same hit rate.
rulest_small_r1.rule · 25.7k
Recovered: 29.06%
Speed: 32,385/h
Efficiency: 1.1294
vs. own lineage: clear jump from rulest_small.rule's 26.23% @ 24,996 — +2.83pp recovered.
vs. similar size: essentially tied with the A1131_25k.rule (29.19% @ 25,000), and ahead of rockyou-30000.rule (26.21% @ 30,000) despite being smaller.
rulest_tiny_r1.rule · 585
Recovered: 9.14%
Speed: 10,186/h
Efficiency: 15.6239
vs. own lineage: more than double rulest_tiny.rule's 4.34% @ 246 — biggest relative gain of the set.
vs. similar size: more than 2x tmesis-specials-num.rule (3.90% @ 588, almost identical size), though still behind concentrator_MT_500.rule (13.54% @ 500) and recondite_675.rule (12.68% @ 675) — room left to climb at this tier.
Takeaway: the tighter depth-8 cap barely costs anything at the large-file end (basically a wash vs. the original run and vs. same-size peers) but pays off increasingly as file size shrinks — medium and small r1 cuts hold their own against well-known reference rulesets of similar size, and tiny_r1 more than doubles both its own predecessor and its closest same-size peer, though it's still not the strongest performer at that tier.
Debug Session Video

A short clip from the debug session, left here as a curiosity — just a few seconds of the hashcat debug pass itself.

Runtime: a few seconds, debug pass only