Skip to content

Conversation

@dogancanbakir
Copy link
Member

@dogancanbakir dogancanbakir commented Dec 3, 2025

Proposed changes

closes #6651

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Summary by CodeRabbit

New Features

  • Added --strict-probe (-sp) CLI flag to enforce strict HTTP probe mode. When enabled, scanning stops if the HTTP probe returns zero URLs, preventing fallback to raw input.

✏️ Tip: You can customize this high-level summary in your review settings.

@dogancanbakir dogancanbakir self-assigned this Dec 3, 2025
@auto-assign auto-assign bot requested a review from dwisiswant0 December 3, 2025 04:44
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Dec 3, 2025

Walkthrough

A new strict probe feature is added to Nuclei via a --strict-probe CLI flag. When enabled, the scanner stops immediately if the internal httpx probe returns zero URLs, skipping fallback to raw input scanning. The feature involves adding an Options field, modifying the HTTP input initialization to return URL counts, and implementing early-exit logic in the enumeration runner.

Changes

Cohort / File(s) Summary
Strict Probe Feature – Configuration & CLI
cmd/nuclei/main.go, pkg/types/types.go
Added StrictProbe boolean field to Options struct. Wired new --strict-probe / -sp CLI flag in optimization flag group, defaulting to false.
Strict Probe Feature – HTTP Input Processing
internal/runner/inputs.go, internal/runner/runner.go
Modified initializeTemplatesHTTPInput() method signature to return an additional int32 URL count. Updated RunEnumeration() to check StrictProbe flag and urlCount; if both indicate no URLs found, logs and returns early without scanning.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

  • Verify that the Copy() method in Options intentionally omits StrictProbe (field will default to false in clones)
  • Confirm early-exit logic in RunEnumeration() correctly prevents further scanning when StrictProbe is enabled and probe finds zero URLs
  • Check that all return paths in initializeTemplatesHTTPInput() properly include the int32 count value

Poem

🐰 A rabbit's flag, so strict and wise,
No fallback schemes, no false surprise!
When httpx finds naught but empty space,
We hop right past, no time to waste.
Swift and clean, the probe stands tall—
Zero results? We skip it all! ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'add strict probe option for httpx' directly and clearly describes the main change—adding a new command-line flag/option for strict probing behavior.
Linked Issues check ✅ Passed The PR implements the core requirement from issue #6651: a --strict-probe flag that stops scanning when httpx probe returns zero URLs, avoiding fallback to raw input.
Out of Scope Changes check ✅ Passed All changes are directly scoped to implementing the strict probe feature: adding the StrictProbe field to Options, wiring it to CLI flags, and integrating the logic into the HTTP input initialization flow.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 6651_add_strict_probe_option

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
internal/runner/runner.go (1)

700-713: Close InputHelper before early-return when StrictProbe finds no URLs

When StrictProbe is true and the httpx probe returns zero URLs, the code returns immediately without closing InputHelper:

if r.options.StrictProbe && urlCount == 0 {
    r.Logger.Info().Msgf("Strict probe mode: No URLs found from httpx probe, skipping scan")
    return nil
}

This leaves the InputHelper (created at line 573) unclosed. The cleanup code at line 757 is only reached in the normal execution path after runStandardEnumeration. To ensure consistent resource cleanup, close the helper before returning:

if r.options.StrictProbe && urlCount == 0 {
    r.Logger.Info().Msgf("Strict probe mode: No URLs found from httpx probe, skipping scan")
+   if executorOpts.InputHelper != nil {
+       _ = executorOpts.InputHelper.Close()
+   }
    return nil
}

Note: The behavior of skipping the entire scan (not just HTTP templates) is intentional per the feature documentation ("no fallback to raw input"), but the resource must still be cleaned up properly.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3dab87b and 39b5880.

📒 Files selected for processing (4)
  • cmd/nuclei/main.go (1 hunks)
  • internal/runner/inputs.go (3 hunks)
  • internal/runner/runner.go (1 hunks)
  • pkg/types/types.go (1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./...

Files:

  • cmd/nuclei/main.go
  • internal/runner/inputs.go
  • internal/runner/runner.go
  • pkg/types/types.go
internal/runner/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Input provider setup should be configured before engine creation for concurrent target processing

Files:

  • internal/runner/inputs.go
  • internal/runner/runner.go
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Lint
🔇 Additional comments (2)
cmd/nuclei/main.go (1)

419-439: CLI wiring for --strict-probe / -sp looks correct

The new optimization flag cleanly binds options.StrictProbe with a sensible description, and -sp does not conflict with existing short flags (-spm is distinct). No issues from a parsing or configuration perspective.

internal/runner/inputs.go (1)

20-87: URL counting and new return signature look correct; be mindful that 0 can mean “no probe run”

The refactor of initializeTemplatesHTTPInput to:

  • allocate a hybrid.HybridMap,
  • atomically count successful ProbeURL results, and
  • return (hm, urlCount, err)

is sound and concurrency-safe (atomic increments in the goroutines, single Load() after swg.Wait()).

One nuance: in the MultiFormatInputProvider branch you now return hm, 0, nil without ever running httpx, so urlCount == 0 can mean either “probe executed and found nothing” or “probe was deliberately skipped for this input type”. The current caller (RunEnumeration) treats urlCount == 0 as “nothing found” when StrictProbe is enabled, so any future callers that add more logic based on urlCount should take this ambiguity into account (e.g., by introducing a sentinel value or a separate “probed” flag if needed).

Comment on lines +204 to +205
// StrictProbe stops scanning when httpx probe returns 0 URLs (no fallback to raw input)
StrictProbe bool
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

StrictProbe is not propagated by Options.Copy()

You’ve added StrictProbe to types.Options but Options.Copy() doesn’t copy it, so any code using a cloned options struct (e.g., authOptions := r.options.Copy()) will always see StrictProbe == false regardless of the original setting. This is inconsistent with how neighboring fields like DisableHTTPProbe and LeaveDefaultPorts are handled and can lead to confusing behavior as usage of Copy() expands.

Recommend including StrictProbe in the copy constructor:

 		DebugRequests:                  options.DebugRequests,
 		DebugResponse:                  options.DebugResponse,
 		DisableHTTPProbe:               options.DisableHTTPProbe,
+		StrictProbe:                    options.StrictProbe,
 		LeaveDefaultPorts:              options.LeaveDefaultPorts,

Also applies to: 476-686

🤖 Prompt for AI Agents
In pkg/types/types.go around lines 204-205, Options.Copy() does not propagate
the new StrictProbe bool field so copies always have StrictProbe==false; update
the copy constructor(s) (including the other copy blocks referenced around lines
476-686) to set StrictProbe on the returned struct from the receiver (i.e.,
include StrictProbe: o.StrictProbe alongside DisableHTTPProbe/LeaveDefaultPorts
etc.) so cloned Options preserve the original StrictProbe value.

Copy link
Member

@dwisiswant0 dwisiswant0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my view, the proposal doesn't fully address the broader set of scenarios. By introducing -strict-probe, we would effectively enforce that all inputs MUST be valid HTTP/S targets by default, which creates a significant limitation because our ecosystem isn't limited to HTTP-protocol-based templates. We have DNS, JavaScript, and Network protocols too, each of which has its own expected input formats.

With -strict-probe enabled, these non-HTTP/S templates would end up being skipped entirely, which breaks their intended functionality and reduces coverage. This effectively narrows the scope of the scanner and could mislead users by silently skipping entire (set of) protocols of templates.
Because of this, I think the proposal doesn’t sufficiently address the full operational context and needs mature design consideration.

@dogancanbakir
Copy link
Member Author

@dwisiswant0 This optional flag allows users to decide whether to keep scanning the specified target, based on the output of the httpx integration as stated. We might need to make it a bit more explicit, maybe -httpx-strict-probe or something similar, but this is not the default behavior.

@dwisiswant0
Copy link
Member

@dogancanbakir - what I was trying to say is, even if a target cannot be probed by httpx, it should still remain eligible for scanning through other non-HTTP protocols. So the inability to probe via httpx shouldn't automatically exclude it from the rest of the non-HTTP-protocol-based templates.

@dwisiswant0
Copy link
Member

and

By introducing -strict-probe, we would effectively enforce that all inputs MUST be valid HTTP/S targets by default

By "default", I mean the flag's standard behavior when users enable it.

@dwisiswant0
Copy link
Member

dwisiswant0 commented Dec 3, 2025

Proposal:

However, Nuclei currently triggers an Auto Fallback mechanism. It ignores the failed probe and proceeds to scan the target using the raw input, assuming it might be a valid HTTP service. In scenarios with large attack surfaces containing many non-HTTP open ports, this causes Nuclei to waste significant time sending thousands of HTTP requests to non-HTTP ports

The way I see that, with -strict-probe, targets that fail httpx probing should be excluded specifically from HTTP-protocol-based templates only, while remaining eligible for all non-HTTP-protocol scans.

But, additionally, that approach would end up skipping any HTTP-protocol-based template that overrides its own host & port settings.

$ grep -Pnr "\{\{Host(name)?\}\}:[0-9]{1,5}" http/
http/vulnerabilities/unifi/unifi-nfc-credentials.yaml:20:        @Host: {{Hostname}}:9780
http/vulnerabilities/unifi/unifi-create-user.yaml:23:        @Host: {{Hostname}}:9780
http/cves/2025/CVE-2025-52665.yaml:55:        @Host: {{Hostname}}:9780
http/cves/2019/CVE-2019-8451.yaml:41:      url=https://{{Host}}:443@{{interactsh-url}}
http/cves/2024/CVE-2024-6396.yaml:54:        @Host: http://{{Host}}:43800
http/cves/2024/CVE-2024-6396.yaml:56:        Host: {{Host}}:43800
http/cves/2018/CVE-2018-8024.yaml:33:      - "{{Host}}:4040/jobs/?\"'><script>alert(document.domain)</script>"
http/cves/2020/CVE-2020-9480.yaml:50:            "spark.master": "spark://{{Hostname}}:6066"

@dogancanbakir
Copy link
Member Author

Proposal:

However, Nuclei currently triggers an Auto Fallback mechanism. It ignores the failed probe and proceeds to scan the target using the raw input, assuming it might be a valid HTTP service. In scenarios with large attack surfaces containing many non-HTTP open ports, this causes Nuclei to waste significant time sending thousands of HTTP requests to non-HTTP ports

The way I see that, with -strict-probe, targets that fail httpx probing should be excluded specifically from HTTP-protocol-based templates only, while remaining eligible for all non-HTTP-protocol scans.

But, additionally, that approach would end up skipping any HTTP-protocol-based template that overrides its own host & port settings.

$ grep -Pnr "\{\{Host(name)?\}\}:[0-9]{1,5}" http/
http/vulnerabilities/unifi/unifi-nfc-credentials.yaml:20:        @Host: {{Hostname}}:9780
http/vulnerabilities/unifi/unifi-create-user.yaml:23:        @Host: {{Hostname}}:9780
http/cves/2025/CVE-2025-52665.yaml:55:        @Host: {{Hostname}}:9780
http/cves/2019/CVE-2019-8451.yaml:41:      url=https://{{Host}}:443@{{interactsh-url}}
http/cves/2024/CVE-2024-6396.yaml:54:        @Host: http://{{Host}}:43800
http/cves/2024/CVE-2024-6396.yaml:56:        Host: {{Host}}:43800
http/cves/2018/CVE-2018-8024.yaml:33:      - "{{Host}}:4040/jobs/?\"'><script>alert(document.domain)</script>"
http/cves/2020/CVE-2020-9480.yaml:50:            "spark.master": "spark://{{Hostname}}:6066"

This is the desired behavior explicitly requested by the community user, and I don't see any issue with this, as long as we don't change nuclei's default behavior.

@dwisiswant0
Copy link
Member

dwisiswant0 commented Dec 3, 2025

Tagging the requester @sixteen250. I want to elaborate a bit more on the concern I'm trying to highlight, because the current behavior of the patch can lead to unintended side effects across the entire scanning flow. From what I understand, the logic effectively works like this:

target input (e.g., host:22)
       ↓
strict-probe applied
       ↓
httpx runs → returns 0 (no successful probe)
       ↓
the target is excluded from scanning (dns, javascript, network, etc.)

The issue here is that strict-probe ends up interpreting an HTTP(S)-level failure as a total host failure. In reality, the only thing that's unreachable is the HTTP protocol surface (typically port 80/443), but other protocols may still be perfectly valid and scannable.

This logic cascades too broadly, causing the scanner to skip ALL protocols simply because httpx cannot probe one or two ports. As a result, templates for DNS, Network, JavaScript, etc., protocols never get a chance to run, even though the host itself might still be very much alive on other ports/services.

tl;dr; This strict-probe behavior becomes overly authoritative: it concludes "the host is dead" rather than "the HTTP surface is dead". This distinction matters because the former shuts down the entire scan scope, while the latter should only limit HTTP-protocol-based templates.

That's the intent behind my concern, defining the boundaries of what strict-probe should and should not influence. Let me know what you think or if clarification is needed.

@dwisiswant0
Copy link
Member

dwisiswant0 commented Dec 3, 2025

To make the scenario easier to visualize, let's use the example provided by the requester, @sixteen250, which uses port 3306 (MySQL).

If the user supplies a target like host:3306 and strict-probe is enabled, then under the current patch logic, that target will inevitably be dropped entirely. The reason is simple: httpx will try to probe it, fail (because 3306 is not an HTTP service), and return no results, which the strict-probe logic interprets as a signal to exclude the target from ALL templates.

But that behavior is incorrect, because while the target is not running an HTTP service, we DO have a number of templates that are specifically intended to scan MySQL on port 3306. Ex:

javascript/misconfiguration/mysql/mysql-empty-password.yaml
javascript/default-logins/mysql-default-login.yaml
javascript/enumeration/mysql/mysql-info.yaml
javascript/enumeration/mysql/mysql-show-variables.yaml
javascript/enumeration/mysql/mysql-user-enum.yaml
javascript/enumeration/mysql/mysql-show-databases.yaml

Those templates are explicitly designed to interact with MySQL, meaning they remain completely valid even if httpx cannot probe the host. So dropping the target solely because httpx fails ends up preventing a whole set of relevant, non-HTTP-protocol-based templates from running.

That's the core of the issue: strict-probe should restrict HTTP-protocol-based templates when httpx fails, and it should NOT disqualify the target from all other protocol templates that operate independently of HTTP.

@Ice3man543
Copy link
Member

Yeah i agree, disabling all templates if http is not detected would be overly strict, we instead we should still check other port templates / protocols etc cc @tarunKoyalwar

@sixteen250
Copy link

Thanks for looping me in @dwisiswant0.

Regarding the naming proposed by @dogancanbakir: I agree that renaming it to -httpx-strict-probe (or similar) makes it much more explicit that this flag controls the behavior of the httpx integration specifically.

Regarding the logic concern: I strongly agree with @dwisiswant0's assessment. My original issue was indeed about the inefficiency and timeouts caused by sending HTTP requests to non-HTTP ports. However, completely dropping the target if the probe fails would lead to missed vulnerabilities on non-Web services (like the MySQL example mentioned).

The ideal behavior for this flag would be: If httpx probe fails → Skip ONLY http protocol templates → Continue scanning with network/javascript/dns templates.

This would solve the timeout performance issue without sacrificing coverage for other services.

@dogancanbakir
Copy link
Member Author

Thanks for the clarification @sixteen250. I got the impression that the desired behavior was the completely skip the target from the following statement:

I would like a new flag (e.g., -strict-probe or -no-fallback) that changes this behavior. If the internal httpx probe returns 0 URLs (meaning the target is confirmed not to be a Web Service), Nuclei should immediately stop processing that host and not fallback to raw input scanning.

Anyway, I'll make necessary changes to the PR.

Thanks @dwisiswant0 and all for the input!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Add a flag (e.g., -strict-probe) to stop scanning if internal httpx probe fails (No Fallback)

5 participants