feat: 引入 so-novel submodule 并提供非交互下载工具
- vendor/so-novel: 上游 v1.12.0 (规则驱动的书源抓取/下载引擎) - tools/src: ToolMain 封装 search/download/toc/sources，stdout 只输出 JSON - tools/build.ps1: 自动定位/下载 JDK21，按平台显式激活 javet 原生库 profile - bin/novel.ps1|.cmd: 命令行入口（人读输出或 -Json） - SKILL.md/README.md: 使用说明与下载原理分析
This commit is contained in:
cheney
+16
@@ -0,0 +1,16 @@
|
|||||||
|
# build artifacts / local runtime
|
||||||
|
/tools/runtime/
|
||||||
|
/tools/build/
|
||||||
|
/tools/.jdk/
|
||||||
|
|
||||||
|
# downloaded books and logs
|
||||||
|
/books/
|
||||||
|
/logs/
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# editor noise
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.iml
|
||||||
|
Thumbs.db
|
||||||
|
.DS_Store
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
[submodule "vendor/so-novel"]
|
||||||
|
path = vendor/so-novel
|
||||||
|
url = https://github.com/freeok/so-novel
|
||||||
@@ -0,0 +1,253 @@
|
|||||||
|
# ebook-skill
|
||||||
|
|
||||||
|
把开源项目 [so-novel](https://github.com/freeok/so-novel)（通用网页小说抓取器）以 **git submodule** 形式引入，
|
||||||
|
并在它外面包一层**非交互 CLI**，让 agent / 脚本可以直接「搜索 → 选源 → 下载 → 拿到文件」，
|
||||||
|
不部署 Web 服务、不跑 TUI 交互菜单。
|
||||||
|
|
||||||
|
```
|
||||||
|
ebook-skill/
|
||||||
|
├─ vendor/so-novel/ # git submodule，上游引擎（Java 21 + Maven）
|
||||||
|
├─ tools/
|
||||||
|
│ ├─ src/skills/novel/ToolMain.java # 封装层：search / download / toc / sources
|
||||||
|
│ ├─ build.ps1 # 构建引擎 jar + 编译封装层 + 生成运行时配置
|
||||||
|
│ ├─ lib.ps1 # 公共函数（定位 JDK21、路径、jar 工具）
|
||||||
|
│ ├─ config.ini.template # 运行时配置模板
|
||||||
|
│ └─ runtime/ # 生成物（gitignore）：app.jar、rules/、config.ini、logs/
|
||||||
|
├─ bin/
|
||||||
|
│ ├─ novel.ps1 # 人读表格 + JSON 两种输出
|
||||||
|
│ └─ novel.cmd # cmd.exe 入口（自动选 pwsh / powershell）
|
||||||
|
├─ books/ # 下载落盘目录（gitignore）
|
||||||
|
├─ SKILL.md # 给 agent 的使用说明
|
||||||
|
└─ README.md
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
git submodule update --init --recursive # 首次
|
||||||
|
pwsh tools\build.ps1 # 首次构建（可自动下载 JDK 21）
|
||||||
|
|
||||||
|
pwsh bin\novel.ps1 sources # 看书源
|
||||||
|
pwsh bin\novel.ps1 search 诡秘之主 # 聚合搜索
|
||||||
|
pwsh bin\novel.ps1 toc <详情页链接> # 只解析目录，验证完整性
|
||||||
|
pwsh bin\novel.ps1 download <详情页链接> -Ext txt
|
||||||
|
```
|
||||||
|
|
||||||
|
依赖：JDK 21（没有则构建脚本从清华镜像下载便携版）、Maven（仅构建时需要）、Git。
|
||||||
|
下载成品在 `books/书名(作者).txt|epub|html|pdf`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 一、so-novel 是怎么下载小说的
|
||||||
|
|
||||||
|
上游不是「针对某个站点写死的爬虫」，而是**规则驱动 + 通用抓取引擎**：
|
||||||
|
每个书源是一段 JSON（`vendor/so-novel/bundle/rules/main.json` 等），引擎按规则抓页面。
|
||||||
|
|
||||||
|
### 1. 书源规则（Rule）
|
||||||
|
|
||||||
|
一条规则对应一个站点，字段分五段：
|
||||||
|
|
||||||
|
| 段 | 作用 | 关键字段 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `url` / `name` / `disabled` / `needProxy` | 站点标识 | `url` 同时是「按详情页 URL 前缀路由到书源」的依据 |
|
||||||
|
| `search` | 搜索接口 | `url`(支持 `%s` 或 `@js:`)、`method`/`data`(GET/POST 表单)、`result`(结果条目选择器)、`bookName`/`author`/`latestChapter`…、`nextPage` |
|
||||||
|
| `book` | 详情页 | `bookName`/`author`/`intro`/`coverUrl`/`latestChapter`/`lastUpdateTime`/`status` |
|
||||||
|
| `toc` | 目录页 | `url`(目录可能不在详情页)、`list`(先取容器 HTML)、`item`(章节条目)、`desc`(倒序)、`nextPage`(分页) |
|
||||||
|
| `chapter` | 正文页 | `content`、`nextPage`(正文分页)、`nextPageInJs`、`nextChapterLink`(末页判定正则)、`filterTxt`/`filterTag`(广告清洗) |
|
||||||
|
| `crawl` | 站点级抓取参数 | `concurrency`、`min/maxInterval`、`maxAttempts` 等覆盖全局配置 |
|
||||||
|
|
||||||
|
选择器同时支持 **CSS** 与 **XPath**（以 `/`、`//`、`(` 开头判定为 XPath），
|
||||||
|
`元素选择器@href` / `@src` / `@content` 表示取属性；`meta[property="og:novel:book_name"]` 这类缺省值让「标准 meta」站点无需逐字段配置。
|
||||||
|
|
||||||
|
选择器后面还能挂 **DSL**：`选择器@js:代码` / `@java:代码` / 多段串联（`a@js:...;@js:...`）。
|
||||||
|
`HtmlExtractor` 先用选择器取原始文本，再交给 `StepEngine` 依次执行 JS（Javet V8 引擎池）或 Java 代码做二次加工
|
||||||
|
——这也是 `javet` 依赖和启动时预热 V8 的原因。
|
||||||
|
|
||||||
|
### 2. 搜索（SearchParser）
|
||||||
|
|
||||||
|
1. 用 `search.url` 拼搜索地址：普通 `%s` 字符串格式化，或 `@js:` 动态生成（签名、加密参数等）；
|
||||||
|
2. GET/POST（`data` 是 JSON，其中 `%s` 按顺序填参数），带随机 UA + Referer，可选 Cookie；
|
||||||
|
3. `result` 选出结果条目，逐条提取书名/作者/最新章节等，链接取 `@href` 的绝对地址；
|
||||||
|
4. 若有 `nextPage`，一次取全部翻页链接并发补齐（去重合并），再按 `search-limit` 截断；
|
||||||
|
5. 若结果条目为空但页面本身是详情页，则退化为「把搜索页当详情页」处理（部分源会直接跳转）；
|
||||||
|
6. 聚合搜索（`AggregatedSearchAction`）对**所有可搜索书源**并发跑一遍，最后 `SearchResultsHandler`
|
||||||
|
用字符串相似度（`StrUtil.similar`）判断关键字更像书名还是作者，过滤低分（<0.25）、按相似度排序。
|
||||||
|
7. 遇 Cloudflare 挑战页（标题白名单判定）时可走外部 `cf-bypass` 服务取真实 HTML。
|
||||||
|
|
||||||
|
### 3. 详情 + 目录（BookParser / TocParser）
|
||||||
|
|
||||||
|
- 详情页解析书名、作者、简介、封面、最新章节、更新时间；规则没写的字段回退到 `og:novel:*` meta。
|
||||||
|
- 目录页并非总在详情页：`toc.url` 可用 `%s` 填详情页里的书籍 id（`book.url` 正则提取）。
|
||||||
|
- 目录分页有两种形态：`nextPage` 命中 `<option>` 下拉（一次性拿到所有分页 URL），或「下一页」按钮递归跟随。
|
||||||
|
- 多目录页用 5 个虚拟线程并发抓取，再按页序拼成章节列表；`desc` 表示站点是倒序，需要反转。
|
||||||
|
|
||||||
|
### 4. 正文抓取（ChapterParser / Crawler）
|
||||||
|
|
||||||
|
- **章节级并发**：Java 21 虚拟线程 + `VirtualThreadLimiter`，默认并发 `min(50, 章节数)`（配置 `concurrency` 可改，上限 100）。
|
||||||
|
- 每章请求前 `sleep` 一个随机间隔（默认 200–400ms，站点规则可覆盖），随机 UA + Referer。
|
||||||
|
- 正文页有分页时按 `nextPage` 循环拼接，用 `nextChapterLink` 正则 / 通用「_数字.html + 下一章文案」判定末页。
|
||||||
|
- 失败重试 `max-retries`（默认 3）次，重试间隔在 `retry-min/max-interval` 内随机并逐次递增；
|
||||||
|
重试用尽则记录日志跳过该章（`enable-retry=0` 时直接中断）。
|
||||||
|
- 每章先落成独立文件（`0001_章节名.html` 等），避免长任务中途失败全丢。
|
||||||
|
|
||||||
|
### 5. 清洗与合成
|
||||||
|
|
||||||
|
`ChapterFilter`：去不可见字符/私用区字符、去 HTML 实体、按 `filterTxt` 正则和 `filterTag` 选择器删广告、
|
||||||
|
去正文开头重复的章节名、`1. 标题` 归一成 `第1章 标题`。
|
||||||
|
`ChapterFormatter`：把 `<br>` 等转成 `<p>` 段落。
|
||||||
|
之后按格式合成：
|
||||||
|
|
||||||
|
| 格式 | 处理器 | 产物 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `txt` | `TxtMergeHandler` | 单文件，开头书名/作者/简介，正文全角缩进；可用 `txt-encoding` 指定 GBK 等 |
|
||||||
|
| `epub` | `EpubMergeHandler` | 带封面、metadata、分章的 epub |
|
||||||
|
| `html` | `HtmlTocHandler` | 带目录的单页 HTML |
|
||||||
|
| `pdf` | `PdfMergeHandler` | openhtmltopdf 生成 PDF |
|
||||||
|
|
||||||
|
中间章节目录默认在合成后删除（`preserve-chapter-cache=1` 可保留）。
|
||||||
|
输出命名：中间目录 `书名 (作者) TXT`，成品 `书名(作者).txt`，均在 `download-path` 下。
|
||||||
|
|
||||||
|
### 6. 反爬与网络层
|
||||||
|
|
||||||
|
随机 UA（`RandomUA`）、Referer、随机间隔、限流重试、可选 HTTP 代理（`[proxy]`）、
|
||||||
|
起点 Cookie（`[cookie] qidian`，用于拿最新/正版章节）、Cloudflare 时借助 `cf-bypass` 服务。
|
||||||
|
`Main` 在启动时创建并销毁一个 V8 runtime，绕开 Windows 上 Javet 首次创建 SIGSEGV 的问题。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 二、这层封装做了什么（以及为什么需要）
|
||||||
|
|
||||||
|
上游三种形态都不适合当工具：
|
||||||
|
|
||||||
|
| 形态 | 问题 |
|
||||||
|
| --- | --- |
|
||||||
|
| TUI（默认） | 交互菜单，无法脚本化 |
|
||||||
|
| Web（`-Dmode=web`，端口 7765） | 需要常驻服务，与「不部署服务」冲突 |
|
||||||
|
| CLI（`-u <详情页>`） | **只能下已知 URL 的单本**，没有搜索；输出是彩色文本，无机器可读结果 |
|
||||||
|
|
||||||
|
所以 `tools/src/skills/novel/ToolMain.java` 直接复用上游库类，补上缺的部分：
|
||||||
|
|
||||||
|
- `search`：并发调用各书源 `SearchParser`，复用上游的相似度过滤/排序，输出 JSON；附带每个源的失败统计。
|
||||||
|
- `download`：与原 CLI 同一条路径（`SourceUtils.getRule(url)` 路由书源 → `BookParser` → `Crawler`），
|
||||||
|
但支持 `--out` 指定目录、返回成品文件路径与大小。
|
||||||
|
- `toc`：只解析目录并给首尾章节，用来**先验证书源质量**（章节数、是否缺章）再决定下载。
|
||||||
|
- `sources`：列出激活书源（id、主页、是否支持搜索）。
|
||||||
|
|
||||||
|
工程细节：
|
||||||
|
|
||||||
|
- **stdout 只有一份 JSON**：库的彩色日志在进程内被重定向截留，`--log` 时以 `logTail` 返回最后 40 行，
|
||||||
|
便于排查「源站 403 / Cloudflare / 目录为空」。
|
||||||
|
- **JSON 中非 ASCII 全部转成 `\uXXXX`**：Windows 控制台代码页（936）不会把中文搞成乱码，
|
||||||
|
`ConvertFrom-Json` / Python `json.loads` 都会还原成正确文本。
|
||||||
|
- **配置即参数**：以 `-Dconfig.file` 指向生成的 `tools/runtime/config.ini`，书源规则走绝对路径，
|
||||||
|
与当前工作目录无关；日志落 `tools/runtime/logs/`。
|
||||||
|
- **首次使用自动构建**：`bin/novel.ps1` 发现 `app.jar` 或 class 缺失会先调 `tools/build.ps1`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 三、命令参考
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# 搜索（-Limit 每个源最多取多少条，默认 30；-Source 只搜某个源；-Timeout 秒）
|
||||||
|
pwsh bin\novel.ps1 search 关键词 [-Limit 30] [-Source 8] [-Timeout 180] [-Log] [-Json]
|
||||||
|
|
||||||
|
# 下载（-Ext txt|epub|html|pdf，默认 txt；-Out 指定目录；URL 也可用 -Url）
|
||||||
|
pwsh bin\novel.ps1 download <详情页链接> [-Ext txt] [-Out D:\books] [-Log] [-Json]
|
||||||
|
|
||||||
|
# 只解析目录
|
||||||
|
pwsh bin\novel.ps1 toc <详情页链接> [-Json]
|
||||||
|
|
||||||
|
# 书源列表
|
||||||
|
pwsh bin\novel.ps1 sources [-Json]
|
||||||
|
```
|
||||||
|
|
||||||
|
`-Json` 输出契约（节选）：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
"command": "search",
|
||||||
|
"keyword": "诡秘之主",
|
||||||
|
"count": 2,
|
||||||
|
"results": [
|
||||||
|
{ "index": 1, "sourceId": 10, "sourceName": "笔趣阁52", "bookName": "诡秘之主",
|
||||||
|
"author": "爱潜水的乌贼", "latestChapter": "新书已发", "lastUpdateTime": "09-26",
|
||||||
|
"url": "https://www.52bqg.com/book_2279/" }
|
||||||
|
],
|
||||||
|
"sources": { "total": 13, "returned": 20, "empty": 3, "failed": [] },
|
||||||
|
"tookMs": 12000
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```json
|
||||||
|
{ "ok": true, "command": "download", "bookName": "诡秘之主", "author": "爱潜水的乌贼",
|
||||||
|
"sourceId": 10, "sourceName": "笔趣阁52", "ext": "txt",
|
||||||
|
"file": "D:\\workbench\\ebook-skill\\books\\诡秘之主(爱潜水的乌贼).txt",
|
||||||
|
"bytes": 1234567, "seconds": 42.1 }
|
||||||
|
```
|
||||||
|
|
||||||
|
失败时 `ok=false` 并带 `error`（附 `--log` 可看上游日志尾部）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 四、运行时配置（`tools/runtime/config.ini`）
|
||||||
|
|
||||||
|
构建时由 `tools/config.ini.template` 生成，仓库位置变化后删除该文件再跑一次 `tools\build.ps1` 即可重建。
|
||||||
|
|
||||||
|
| 配置 | 说明 |
|
||||||
|
| --- | --- |
|
||||||
|
| `[download] download-path` | 默认 `books/`，可用 `-Out` 单次覆盖 |
|
||||||
|
| `[download] extname` | 默认格式，命令行 `-Ext` 优先 |
|
||||||
|
| `[download] txt-encoding` | txt 编码，老设备可设 `GBK` |
|
||||||
|
| `[download] preserve-chapter-cache` | 1 = 保留中间章节文件 |
|
||||||
|
| `[source] active-rules` | 激活的规则文件（默认 `runtime/rules/main.json`，13 个源） |
|
||||||
|
| `[source] search-limit` | 每个源搜索结果上限 |
|
||||||
|
| `[source] search-filter` | 1 = 按相似度过滤低质量结果 |
|
||||||
|
| `[crawl] concurrency` | 章节并发，空 = 自动（≤50，上限 100） |
|
||||||
|
| `[crawl] min/max-interval` | 章节请求随机间隔（毫秒） |
|
||||||
|
| `[crawl] max-retries` | 单章重试次数 |
|
||||||
|
| `[proxy]` | 需要代理才能访问的书源：`enabled=1` + `host/port` |
|
||||||
|
| `[cookie] qidian` | 起点 `w_tsfp`，用于部分源的最新章节 |
|
||||||
|
| `[global] cf-bypass` | Cloudflare 绕过服务地址（如 `https://github.com/sarperavci/CloudflareBypassForScraping` 部署的实例） |
|
||||||
|
|
||||||
|
`runtime/rules/` 下的其他规则包（`cloudflare.json`、`proxy-required.json`、`rate-limit.json`、`no-search.json`）
|
||||||
|
默认**未激活**：需要时把 `active-rules` 指向对应文件（或把这些源合并进一个 json）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 五、维护
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
# 更新上游代码（GitHub 直连不通时，本仓库已在 .git/config 配了 ghfast.top 镜像 insteadOf）
|
||||||
|
git -C vendor/so-novel fetch --tags
|
||||||
|
git -C vendor/so-novel checkout <tag>
|
||||||
|
git add vendor/so-novel && git commit -m "bump so-novel"
|
||||||
|
|
||||||
|
# 重建（引擎变更时必须；规则包更新可只刷规则）
|
||||||
|
pwsh tools\build.ps1 -Force -RefreshRules
|
||||||
|
```
|
||||||
|
|
||||||
|
已知坑：
|
||||||
|
|
||||||
|
- **Javet 原生库**：so-novel 依赖 JS 规则（`@js:`），需要平台对应的 `javet-v8-*` 原生库。
|
||||||
|
它的 Maven profile 用 `x86_64` 匹配架构，而 Windows JDK 报 `amd64`，自动激活会失效，
|
||||||
|
症状是「Javet library /libjavet-v8-windows-x86_64...dll is not found」、依赖 JS 规则的书源全部空结果。
|
||||||
|
`tools/build.ps1` 已按平台显式传 `-P windows-x64|linux-x64|…`，重建后 `app.jar` 约 36 MB（含原生库）。
|
||||||
|
- **GitHub 被重置**：仓库本地配置了 `url."https://ghfast.top/https://github.com/".insteadOf`，
|
||||||
|
删除用 `git config --unset url."https://ghfast.top/https://github.com/".insteadOf`。
|
||||||
|
- **JDK 21 必需**（虚拟线程 + 上游 class 版本 65）；`Find-Jdk21` 会依次找
|
||||||
|
`EBOOK_NOVEL_JAVA_HOME` → `JAVA_HOME` → `tools/.jdk` → 常见安装路径（含 IntelliJ 自带 JBR）。
|
||||||
|
- **Maven** 只在构建时需要；Maven Central 走内网 nexus / 阿里云镜像均可。
|
||||||
|
- **规则失效**是最常见的失败原因（站点改版/关站）：`search` 出来为空、`toc` 章节数异常、
|
||||||
|
下载 txt 只有简介——此时换一个书源，或更新子模块/规则。
|
||||||
|
- **目录质量比源数量重要**：`toc` 里首章不是第 1 章（分页缺失）或顺序反了（站点倒序但规则没标 `desc`），
|
||||||
|
都会产出残书/反序书，直接换源。
|
||||||
|
- **别把并发调太高**：这些站点普遍限流，`concurrency` 40–60、间隔 200–400ms 是上游默认值，够用。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 免责声明
|
||||||
|
|
||||||
|
本项目仅用于个人学习与技术研究，请遵守目标站点服务条款与著作权法规，不要用于商业分发。
|
||||||
@@ -0,0 +1,94 @@
|
|||||||
|
---
|
||||||
|
name: ebook-novel
|
||||||
|
description: 用 so-novel 引擎聚合搜索多个中文小说书源并整本下载（txt/epub/html/pdf）。当用户要“找某本小说/下载小说/离线看/换个书源/转 epub/txt”时使用；本技能是本地 CLI 工具，不启动任何常驻服务。
|
||||||
|
---
|
||||||
|
|
||||||
|
# 小说搜索与下载（so-novel）
|
||||||
|
|
||||||
|
本仓库把 [so-novel](https://github.com/freeok/so-novel) 作为 submodule 内嵌，并用 `bin\novel.ps1`
|
||||||
|
封装成非交互 CLI。**不要**启动 so-novel 的 Web 模式（端口 7765），也**不要**去它的 TUI 菜单里操作。
|
||||||
|
|
||||||
|
## 什么时候用
|
||||||
|
|
||||||
|
- 用户给出书名/作者，想找可下载的书源或直接拿到整本文件；
|
||||||
|
- 需要特定格式（txt / epub / html / pdf），或需要按章节验证书源是否完整；
|
||||||
|
- 已有详情页链接，只想下载或看目录。
|
||||||
|
|
||||||
|
不适用：正版/付费平台（起点、微信读书等）的授权内容；无对应书源的外文小说。
|
||||||
|
|
||||||
|
## 标准流程
|
||||||
|
|
||||||
|
1. **搜索**（并发查全部激活书源，约 5–20 秒）
|
||||||
|
```powershell
|
||||||
|
pwsh bin\novel.ps1 search "<书名或作者>" -Limit 10 -Json
|
||||||
|
```
|
||||||
|
取 JSON 的 `results[]`（按相似度排序）：`bookName` / `author` / `sourceName` / `url` / `latestChapter`。
|
||||||
|
|
||||||
|
2. **挑源**（关键，别拿到第一条就下）
|
||||||
|
- 书名、作者与用户要求一致（注意同名续作、同人：「诡秘之主：瑶光」不是「诡秘之主」）；
|
||||||
|
- 同一本书多个源时，用 `toc` 比较章节数与首尾章节，取**章节最多、更新最近**的源；
|
||||||
|
- 若某源 `latestChapter` 明显落后（几个月前），换源。
|
||||||
|
|
||||||
|
3. **验证目录**（可选但推荐，目录页反爬/改版会导致下出空书或残书）
|
||||||
|
```powershell
|
||||||
|
pwsh bin\novel.ps1 toc "<详情页链接>" -Json
|
||||||
|
```
|
||||||
|
看 `chapters` 数量和 `firstChapters` / `lastChapters`，两个常见异常信号：
|
||||||
|
- **首章不是「第1章/序章」**（例如直接从“第101章”开始）→ 目录分页没抓全，会下出残书，换源；
|
||||||
|
- **开头是「后记/尾声」、结尾是「第1章」** → 站点倒序且规则没声明 `desc`，成品章节会反序，换源。
|
||||||
|
|
||||||
|
4. **下载**
|
||||||
|
```powershell
|
||||||
|
pwsh bin\novel.ps1 download "<详情页链接>" -Ext txt -Json
|
||||||
|
```
|
||||||
|
默认落盘 `<仓库>\books\书名(作者).txt`；`-Ext epub` 生成带封面的 epub；`-Out` 可换目录。
|
||||||
|
返回的 `file` 就是最终成品路径，`chapters` 是章节数，`ok=false` 时读 `error`。
|
||||||
|
|
||||||
|
5. **汇报**：给出书名、作者、来源站点、章节量级、文件绝对路径和大小；需要时说明换过几个源和原因。
|
||||||
|
|
||||||
|
## 输出契约
|
||||||
|
|
||||||
|
所有命令都是**一份 JSON**（`-Json`）或人读表格（默认）：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"ok":true,"command":"download","bookName":"...","author":"...",
|
||||||
|
"sourceId":8,"sourceName":"笔趣阁365","ext":"txt",
|
||||||
|
"file":"...\\books\\书名(作者).txt","bytes":1870447,"seconds":4.51}
|
||||||
|
```
|
||||||
|
|
||||||
|
- 中文字段在 JSON 里是 `\uXXXX` 转义，解析后即为正确文本（这是为了绕开 Windows 控制台代码页）。
|
||||||
|
- `search` 的 `sources.failed[]` 列出抓取失败的源及错误（`403`、超时等），`sources.empty` 是返回空的源数；
|
||||||
|
这两个数字异常大说明网络/规则整体有问题，而不是这本书没有。
|
||||||
|
|
||||||
|
## 失败处理顺序
|
||||||
|
|
||||||
|
1. `search` 命中 0：换更完整的书名/作者，`-Limit` 调大，再试一次（不同源索引延迟不同）。
|
||||||
|
2. 命中的 URL 下载失败（`ok=false` 或 `toc` 为空）：**换另一个源的 URL**，不要在同一源重试多次。
|
||||||
|
3. 连续多个源失败：加 `-Log` 看 `logTail` 里的上游报错，区分几种情况：
|
||||||
|
- `403` / `Just a moment...` → Cloudflare 或封 UA，该源需要代理/`cf-bypass`，直接跳过；
|
||||||
|
- 详情页书名或作者为空 → 站点改版，规则失效，跳过；
|
||||||
|
- `源站章节目录为空` → 目录页反爬，跳过。
|
||||||
|
4. 全书章节数正常但个别章节为空：上游允许单章失败跳过，成品仍可用；如实告知用户可能有缺章。
|
||||||
|
|
||||||
|
## 约束与礼貌
|
||||||
|
|
||||||
|
- 单机、串行使用；不要为了「快」把 `[crawl] concurrency` 调到 100，源站会限流并封 IP。
|
||||||
|
- 不要并发跑多个 `download`；同一本书不要反复重下（`books/` 里已有就直接报告路径）。
|
||||||
|
- 不要修改 `vendor/so-novel` 里的代码；需要新能力就加到 `tools/src/skills/novel/ToolMain.java`。
|
||||||
|
- 上游更新后必须重建：`pwsh tools\build.ps1 -Force -RefreshRules`（详见 README 的维护章节）。
|
||||||
|
|
||||||
|
## 一次完整的例子
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
pwsh bin\novel.ps1 search "三体" -Limit 5 -Json
|
||||||
|
# -> results 里 刘慈欣《三体1：地球往事》/ 笔趣阁52 / https://www.52bqg.com/book_40390/
|
||||||
|
|
||||||
|
pwsh bin\novel.ps1 toc "https://www.52bqg.com/book_40390/" -Json
|
||||||
|
# -> chapters: 37，开头「第1章 1科学边界」，结尾「第37章 后记」——顺序正常
|
||||||
|
|
||||||
|
pwsh bin\novel.ps1 download "https://www.52bqg.com/book_40390/" -Ext txt -Json
|
||||||
|
# -> file: D:\workbench\ebook-skill\books\三体1：地球往事(刘慈欣).txt
|
||||||
|
```
|
||||||
|
|
||||||
|
对照：同一本书在「少年小说网」目录是倒序（开头“后记”、结尾“第1章”），下载会得到反序章节；
|
||||||
|
若某个源的目录条数明显偏少、首章又不是第 1 章，说明分页没抓全，都会产出残书——这两种情况都应换源。
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
@echo off
|
||||||
|
rem ebook-skill :: so-novel command line shim for cmd.exe
|
||||||
|
setlocal
|
||||||
|
where pwsh >nul 2>nul
|
||||||
|
if %ERRORLEVEL%==0 (
|
||||||
|
pwsh -NoProfile -ExecutionPolicy Bypass -File "%~dp0novel.ps1" %*
|
||||||
|
) else (
|
||||||
|
powershell -NoProfile -ExecutionPolicy Bypass -File "%~dp0novel.ps1" %*
|
||||||
|
)
|
||||||
|
exit /b %ERRORLEVEL%
|
||||||
+229
@@ -0,0 +1,229 @@
|
|||||||
|
# ebook-skill :: so-novel command line
|
||||||
|
#
|
||||||
|
# novel search <keyword> [-Limit N] [-Source ID] [-Timeout SEC] [-Log] [-Json]
|
||||||
|
# novel download <detail-url> [-Ext txt|epub|html|pdf] [-Out DIR] [-Log] [-Json]
|
||||||
|
# novel toc <detail-url> [-Json]
|
||||||
|
# novel sources [-Json]
|
||||||
|
#
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
. (Join-Path $PSScriptRoot '..\tools\lib.ps1')
|
||||||
|
|
||||||
|
$paths = Get-ToolPaths
|
||||||
|
|
||||||
|
function Show-Help {
|
||||||
|
@"
|
||||||
|
ebook-skill / so-novel
|
||||||
|
|
||||||
|
novel search <关键词> [-Limit N] [-Source ID] [-Timeout SEC] [-Log] [-Json]
|
||||||
|
novel download <详情页链接> [-Ext txt|epub|html|pdf] [-Out 目录] [-Log] [-Json]
|
||||||
|
novel toc <详情页链接> [-Json] 只看目录，不下载
|
||||||
|
novel sources [-Json] 列出内置书源
|
||||||
|
|
||||||
|
说明
|
||||||
|
-Json 输出原始 JSON（默认输出人读表格）
|
||||||
|
-Log 附带 so-novel 运行日志尾部，便于排查源站问题
|
||||||
|
-Limit 每个书源最多取多少条搜索结果（默认 30）
|
||||||
|
下载落盘默认在 <仓库>\books，文件名形如 书名(作者).txt
|
||||||
|
"@ | Write-Host
|
||||||
|
}
|
||||||
|
|
||||||
|
function Ensure-Built {
|
||||||
|
if ((Test-Path $paths.AppJar) -and (Test-Path $paths.Classes)) { return }
|
||||||
|
Write-Host '[setup] first run: building so-novel engine + wrapper ...' -ForegroundColor Yellow
|
||||||
|
& (Join-Path $paths.Tools 'build.ps1')
|
||||||
|
if ($LASTEXITCODE -ne 0) { throw "build failed (exit $LASTEXITCODE)" }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Invoke-Tool {
|
||||||
|
param([string[]]$JavaArgs)
|
||||||
|
Ensure-Built
|
||||||
|
$java = Get-JavaCommand
|
||||||
|
$cp = $paths.AppJar + ';' + $paths.Classes
|
||||||
|
Push-Location $paths.Runtime
|
||||||
|
try {
|
||||||
|
$raw = (& $java '-Dfile.encoding=UTF-8' "-Dconfig.file=$($paths.Config)" '-cp' $cp 'skills.novel.ToolMain' @JavaArgs 2>&1 | Out-String)
|
||||||
|
$code = $LASTEXITCODE
|
||||||
|
} finally {
|
||||||
|
Pop-Location
|
||||||
|
}
|
||||||
|
return [pscustomobject]@{ Raw = $raw.Trim(); Code = $code }
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-Option {
|
||||||
|
param([string[]]$Tokens, [string]$Name, $Default = $null)
|
||||||
|
for ($i = 0; $i -lt $Tokens.Count; $i++) {
|
||||||
|
$t = $Tokens[$i]
|
||||||
|
if ($t -ieq $Name -and ($i + 1) -lt $Tokens.Count) { return $Tokens[$i + 1] }
|
||||||
|
if ($t -like "$Name=*") { return $t.Substring($Name.Length + 1) }
|
||||||
|
}
|
||||||
|
return $Default
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-Flag {
|
||||||
|
param([string[]]$Tokens, [string]$Name)
|
||||||
|
foreach ($t in $Tokens) { if ($t -ieq $Name) { return $true } }
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-Positionals {
|
||||||
|
param([string[]]$Tokens, [string[]]$ValueOptions)
|
||||||
|
$values = @()
|
||||||
|
for ($i = 0; $i -lt $Tokens.Count; $i++) {
|
||||||
|
$t = $Tokens[$i]
|
||||||
|
if ($t.StartsWith('-')) {
|
||||||
|
if ($ValueOptions -contains $t -and ($i + 1) -lt $Tokens.Count) { $i++ }
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
$values += $t
|
||||||
|
}
|
||||||
|
return $values
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-SearchResult {
|
||||||
|
param($Result)
|
||||||
|
if (-not $Result.ok) {
|
||||||
|
Write-Host ("搜索失败: " + $Result.error) -ForegroundColor Red
|
||||||
|
return
|
||||||
|
}
|
||||||
|
$failedCount = 0
|
||||||
|
if ($Result.sources.failed) { $failedCount = @($Result.sources.failed).Count }
|
||||||
|
Write-Host ("关键词「{0}」：命中 {1} 条，书源 {2} 个（空 {3}，异常 {4}），耗时 {5} ms" -f `
|
||||||
|
$Result.keyword, $Result.count, $Result.sources.total, $Result.sources.empty, $failedCount, $Result.tookMs) -ForegroundColor Cyan
|
||||||
|
|
||||||
|
if ($Result.count -eq 0) {
|
||||||
|
Write-Host '没有命中。试试更完整的书名/作者，或 -Limit 调大。' -ForegroundColor Yellow
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
$rows = @()
|
||||||
|
foreach ($r in $Result.results) {
|
||||||
|
$latest = [string]$r.latestChapter
|
||||||
|
if ($latest.Length -gt 24) { $latest = $latest.Substring(0, 24) + '...' }
|
||||||
|
$author = [string]$r.author
|
||||||
|
if (-not $author) { $author = '?' }
|
||||||
|
Write-Host ("[{0}] {1} —— {2}" -f $r.index, $r.bookName, $author) -ForegroundColor White
|
||||||
|
Write-Host (" 最新章节: {0} 更新: {1} 书源: {2}" -f $latest, $r.lastUpdateTime, $r.sourceName) -ForegroundColor DarkGray
|
||||||
|
Write-Host (" {0}" -f $r.url) -ForegroundColor DarkCyan
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Host '下载示例: novel download <链接> -Ext txt' -ForegroundColor DarkGray
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-DownloadResult {
|
||||||
|
param($Result)
|
||||||
|
if (-not $Result.ok) {
|
||||||
|
Write-Host ("下载失败: " + $Result.error) -ForegroundColor Red
|
||||||
|
if ($Result.bookName) { Write-Host (" 书籍: {0} / {1}" -f $Result.bookName, $Result.author) }
|
||||||
|
return
|
||||||
|
}
|
||||||
|
Write-Host ("完成《{0}》（{1}） {2} 章 {3} s" -f $Result.bookName, $Result.author, $Result.chapters, $Result.seconds) -ForegroundColor Green
|
||||||
|
Write-Host (" 文件: {0} ({1:N0} KB)" -f $Result.file, ($Result.bytes / 1KB))
|
||||||
|
Write-Host (" 书源: #{0} {1}" -f $Result.sourceId, $Result.sourceName) -ForegroundColor DarkGray
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-TocResult {
|
||||||
|
param($Result)
|
||||||
|
if (-not $Result.ok) {
|
||||||
|
Write-Host ("目录解析失败: " + $Result.error) -ForegroundColor Red
|
||||||
|
return
|
||||||
|
}
|
||||||
|
Write-Host ("《{0}》 书源 #{1} {2}" -f $Result.url, $Result.sourceId, $Result.sourceName) -ForegroundColor Cyan
|
||||||
|
Write-Host (" 章节数: {0}" -f $Result.chapters)
|
||||||
|
Write-Host (" 开头: " + (@($Result.firstChapters) -join ' / '))
|
||||||
|
Write-Host (" 结尾: " + (@($Result.lastChapters) -join ' / '))
|
||||||
|
}
|
||||||
|
|
||||||
|
function Show-Sources {
|
||||||
|
param($Result)
|
||||||
|
Write-Host ("激活规则: {0} 共 {1} 个书源" -f $Result.activeRules, $Result.count) -ForegroundColor Cyan
|
||||||
|
foreach ($s in $Result.sources) {
|
||||||
|
$searchable = '-'
|
||||||
|
if ($s.searchable) { $searchable = '支持搜索' }
|
||||||
|
Write-Host ("{0,2}. {1,-14} {2,-8} {3}" -f $s.id, $s.name, $searchable, $s.url)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- dispatch
|
||||||
|
function Invoke-SearchCommand {
|
||||||
|
param([string[]]$Tokens)
|
||||||
|
$valueOptions = @('-Limit', '-Source', '-Timeout')
|
||||||
|
$keyword = (@(Get-Positionals -Tokens $Tokens -ValueOptions $valueOptions) -join ' ').Trim()
|
||||||
|
if (-not $keyword) { Write-Host '用法: novel search <关键词>' -ForegroundColor Yellow; exit 2 }
|
||||||
|
$javaArgs = @('search', $keyword,
|
||||||
|
'--limit', [string](Get-Option $Tokens '-Limit' 30),
|
||||||
|
'--source', [string](Get-Option $Tokens '-Source' '0'),
|
||||||
|
'--timeout', [string](Get-Option $Tokens '-Timeout' '180'))
|
||||||
|
if (Test-Flag $Tokens '-Log') { $javaArgs += '--log' }
|
||||||
|
$result = Invoke-Tool -JavaArgs $javaArgs
|
||||||
|
if (Test-Flag $Tokens '-Json') { $result.Raw | Write-Output }
|
||||||
|
else { Show-SearchResult ($result.Raw | ConvertFrom-Json) }
|
||||||
|
exit $result.Code
|
||||||
|
}
|
||||||
|
|
||||||
|
function Invoke-DownloadCommand {
|
||||||
|
param([string[]]$Tokens)
|
||||||
|
$valueOptions = @('-Ext', '-Out', '-Url')
|
||||||
|
$url = Get-Option $Tokens '-Url' $null
|
||||||
|
if (-not $url) {
|
||||||
|
$positionals = @(Get-Positionals -Tokens $Tokens -ValueOptions $valueOptions)
|
||||||
|
if ($positionals.Count -gt 0) { $url = $positionals[0] }
|
||||||
|
}
|
||||||
|
if (-not $url) { Write-Host '用法: novel download <详情页链接> [-Ext txt] [-Out 目录]' -ForegroundColor Yellow; exit 2 }
|
||||||
|
$out = Get-Option $Tokens '-Out' $null
|
||||||
|
if ($out) { $out = (New-Item -ItemType Directory -Force -Path $out).FullName }
|
||||||
|
$javaArgs = @('download', $url, '--ext', [string](Get-Option $Tokens '-Ext' 'txt'))
|
||||||
|
if ($out) { $javaArgs += @('--out', $out) }
|
||||||
|
if (Test-Flag $Tokens '-Log') { $javaArgs += '--log' }
|
||||||
|
$result = Invoke-Tool -JavaArgs $javaArgs
|
||||||
|
if (Test-Flag $Tokens '-Json') { $result.Raw | Write-Output }
|
||||||
|
else { Show-DownloadResult ($result.Raw | ConvertFrom-Json) }
|
||||||
|
exit $result.Code
|
||||||
|
}
|
||||||
|
|
||||||
|
function Invoke-TocCommand {
|
||||||
|
param([string[]]$Tokens)
|
||||||
|
$url = Get-Option $Tokens '-Url' $null
|
||||||
|
if (-not $url) {
|
||||||
|
$positionals = @(Get-Positionals -Tokens $Tokens -ValueOptions @('-Url'))
|
||||||
|
if ($positionals.Count -gt 0) { $url = $positionals[0] }
|
||||||
|
}
|
||||||
|
if (-not $url) { Write-Host '用法: novel toc <详情页链接>' -ForegroundColor Yellow; exit 2 }
|
||||||
|
$javaArgs = @('toc', $url)
|
||||||
|
if (Test-Flag $Tokens '-Log') { $javaArgs += '--log' }
|
||||||
|
$result = Invoke-Tool -JavaArgs $javaArgs
|
||||||
|
if (Test-Flag $Tokens '-Json') { $result.Raw | Write-Output }
|
||||||
|
else { Show-TocResult ($result.Raw | ConvertFrom-Json) }
|
||||||
|
exit $result.Code
|
||||||
|
}
|
||||||
|
|
||||||
|
$argv = @($args)
|
||||||
|
if ($argv.Count -eq 0) { Show-Help; exit 0 }
|
||||||
|
|
||||||
|
$command = ([string]$argv[0]).ToLowerInvariant()
|
||||||
|
$rest = @()
|
||||||
|
if ($argv.Count -gt 1) { $rest = $argv[1..($argv.Count - 1)] }
|
||||||
|
|
||||||
|
switch ($command) {
|
||||||
|
'search' { Invoke-SearchCommand -Tokens $rest }
|
||||||
|
'download' { Invoke-DownloadCommand -Tokens $rest }
|
||||||
|
'dl' { Invoke-DownloadCommand -Tokens $rest }
|
||||||
|
'toc' { Invoke-TocCommand -Tokens $rest }
|
||||||
|
'chapters' { Invoke-TocCommand -Tokens $rest }
|
||||||
|
'sources' {
|
||||||
|
$result = Invoke-Tool -JavaArgs @('sources')
|
||||||
|
if (Test-Flag $rest '-Json') { $result.Raw | Write-Output }
|
||||||
|
else { Show-Sources ($result.Raw | ConvertFrom-Json) }
|
||||||
|
exit $result.Code
|
||||||
|
}
|
||||||
|
'build' {
|
||||||
|
& (Join-Path $paths.Tools 'build.ps1') @rest
|
||||||
|
exit $LASTEXITCODE
|
||||||
|
}
|
||||||
|
default {
|
||||||
|
if ($command -in @('-h', '--help', 'help')) { Show-Help; exit 0 }
|
||||||
|
Write-Host "未知命令: $command" -ForegroundColor Red
|
||||||
|
Show-Help
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
}
|
||||||
+114
@@ -0,0 +1,114 @@
|
|||||||
|
# Builds the so-novel engine jar (from the submodule) and the tool wrapper classes.
|
||||||
|
# Run once after clone / submodule update: pwsh tools\build.ps1
|
||||||
|
param(
|
||||||
|
[switch]$Force, # rebuild so-novel even if runtime\app.jar exists
|
||||||
|
[switch]$RefreshRules # re-copy bundle\rules over runtime\rules
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
. (Join-Path $PSScriptRoot 'lib.ps1')
|
||||||
|
|
||||||
|
$p = Get-ToolPaths
|
||||||
|
Write-Host "[build] project root : $($p.Root)"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 1. JDK 21
|
||||||
|
$jdk = Find-Jdk21 -AllowDownload
|
||||||
|
if (-not $jdk) { throw 'No JDK 21 available and download failed. Set EBOOK_NOVEL_JAVA_HOME.' }
|
||||||
|
Write-Host "[build] JDK 21 : $jdk"
|
||||||
|
$javac = Join-Path $jdk 'bin\javac.exe'
|
||||||
|
if (-not (Test-Path $javac)) { throw "javac not found in $jdk (a JRE is not enough)" }
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 2. so-novel app.jar
|
||||||
|
New-Item -ItemType Directory -Force $p.Runtime | Out-Null
|
||||||
|
if ($Force -or -not (Test-Path $p.AppJar)) {
|
||||||
|
if (-not (Test-Path (Join-Path $p.Vendor 'pom.xml'))) {
|
||||||
|
throw 'vendor/so-novel is empty. Run: git submodule update --init --recursive'
|
||||||
|
}
|
||||||
|
$mvn = Get-Command mvn -ErrorAction SilentlyContinue
|
||||||
|
if (-not $mvn) { throw 'Maven (mvn) is required to build so-novel. Install Maven or use a prebuilt app.jar.' }
|
||||||
|
|
||||||
|
Write-Host '[build] mvn package so-novel ...' -ForegroundColor Cyan
|
||||||
|
# so-novel 用 <os><arch>x86_64</arch> 激活 javet 原生库 profile，而 Windows JDK 报的是 amd64，
|
||||||
|
# 自动激活会失效（导致 JS 规则的书源全部 "Javet library ... not found"），这里显式指定。
|
||||||
|
$platformProfile = switch -Regex ("$([System.Runtime.InteropServices.RuntimeInformation]::OSDescription)|$([System.Runtime.InteropServices.RuntimeInformation]::OSArchitecture)") {
|
||||||
|
'Windows.*X64' { 'windows-x64' }
|
||||||
|
'Linux.*X64' { 'linux-x64' }
|
||||||
|
'Linux.*Arm64' { 'linux-arm64' }
|
||||||
|
'OSX|Darwin.*X64' { 'macos-x64' }
|
||||||
|
'OSX|Darwin.*Arm64' { 'macos-arm64' }
|
||||||
|
default { $null }
|
||||||
|
}
|
||||||
|
if ($platformProfile) {
|
||||||
|
Write-Host "[build] javet profile: $platformProfile" -ForegroundColor Cyan
|
||||||
|
} else {
|
||||||
|
Write-Warning 'Unknown platform, javet native library may be missing (JS-based rules will fail).'
|
||||||
|
}
|
||||||
|
$previousJavaHome = $env:JAVA_HOME
|
||||||
|
$previousPath = $env:PATH
|
||||||
|
try {
|
||||||
|
$env:JAVA_HOME = $jdk
|
||||||
|
$env:PATH = (Join-Path $jdk 'bin') + ';' + $previousPath
|
||||||
|
Push-Location $p.Vendor
|
||||||
|
try {
|
||||||
|
$mvnArgs = @('-B', '-DskipTests', '-Dlaunch4j.skip=true')
|
||||||
|
if ($platformProfile) { $mvnArgs += @('-P', $platformProfile) }
|
||||||
|
$mvnArgs += @('clean', 'package')
|
||||||
|
& $mvn.Source @mvnArgs
|
||||||
|
if ($LASTEXITCODE -ne 0) { throw "mvn package failed with exit code $LASTEXITCODE" }
|
||||||
|
} finally {
|
||||||
|
Pop-Location
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
$env:JAVA_HOME = $previousJavaHome
|
||||||
|
$env:PATH = $previousPath
|
||||||
|
}
|
||||||
|
|
||||||
|
$built = Join-Path $p.Vendor 'target\app-jar-with-dependencies.jar'
|
||||||
|
if (-not (Test-Path $built)) { $built = Join-Path $p.Vendor 'target\app.jar' }
|
||||||
|
if (-not (Test-Path $built)) { throw 'so-novel build produced no jar in target/' }
|
||||||
|
Copy-Item $built $p.AppJar -Force
|
||||||
|
Write-Host "[build] app.jar : $($p.AppJar) ($([math]::Round((Get-Item $p.AppJar).Length / 1MB, 1)) MB)"
|
||||||
|
} else {
|
||||||
|
Write-Host "[build] app.jar : up to date ($([math]::Round((Get-Item $p.AppJar).Length / 1MB, 1)) MB)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 3. rule bundles
|
||||||
|
New-Item -ItemType Directory -Force $p.Rules | Out-Null
|
||||||
|
$sourceRules = Join-Path $p.Vendor 'bundle\rules'
|
||||||
|
foreach ($file in (Get-ChildItem $sourceRules -Filter '*.json*' -File)) {
|
||||||
|
$target = Join-Path $p.Rules $file.Name
|
||||||
|
if ($RefreshRules -or -not (Test-Path $target)) {
|
||||||
|
Copy-Item $file.FullName $target -Force
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Write-Host "[build] rules : $((Get-ChildItem $p.Rules -File).Count) files in $($p.Rules)"
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 4. tool classes
|
||||||
|
if (Test-Path $p.Classes) { Remove-Item $p.Classes -Recurse -Force }
|
||||||
|
New-Item -ItemType Directory -Force $p.Classes | Out-Null
|
||||||
|
$sources = @(Get-ChildItem (Join-Path $p.Tools 'src') -Filter '*.java' -Recurse | ForEach-Object { $_.FullName })
|
||||||
|
Write-Host '[build] javac : tool wrapper ...' -ForegroundColor Cyan
|
||||||
|
& $javac '-encoding' 'UTF-8' '--release' '21' '-cp' $p.AppJar '-d' $p.Classes @sources
|
||||||
|
if ($LASTEXITCODE -ne 0) { throw "javac failed with exit code $LASTEXITCODE" }
|
||||||
|
|
||||||
|
# Optional single-file tool jar (a plain classes dir works too; JBR has no jar.exe).
|
||||||
|
$jarTool = Find-JarTool
|
||||||
|
if ($jarTool) {
|
||||||
|
& $jarTool '--create' '--file' $p.ToolJar '-C' $p.Classes '.'
|
||||||
|
if ($LASTEXITCODE -ne 0) { Write-Warning 'jar packaging failed; the classes directory will be used instead.' }
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- 5. runtime config
|
||||||
|
New-Item -ItemType Directory -Force $p.Books | Out-Null
|
||||||
|
if ($Force -or -not (Test-Path $p.Config)) {
|
||||||
|
$template = Get-Content (Join-Path $p.Tools 'config.ini.template') -Raw
|
||||||
|
$ini = $template.Replace('@BOOKS@', ($p.Books -replace '\\', '/'))
|
||||||
|
$ini = $ini.Replace('@RULES@', ((Join-Path $p.Rules 'main.json') -replace '\\', '/'))
|
||||||
|
[System.IO.File]::WriteAllText($p.Config, $ini, (New-Object System.Text.UTF8Encoding($false)))
|
||||||
|
Write-Host "[build] config.ini : $($p.Config)"
|
||||||
|
} else {
|
||||||
|
Write-Host "[build] config.ini : kept existing ($($p.Config))"
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Host ''
|
||||||
|
Write-Host 'Build finished. Try: pwsh bin\novel.ps1 sources' -ForegroundColor Green
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
[global]
|
||||||
|
# 关闭 GitHub 自动更新检查（内网/无外网环境必须为 0）
|
||||||
|
auto-update = 0
|
||||||
|
gh-proxy =
|
||||||
|
cf-bypass =
|
||||||
|
|
||||||
|
[download]
|
||||||
|
# 默认落盘目录，由 tools/build.ps1 按仓库位置生成
|
||||||
|
download-path = @BOOKS@
|
||||||
|
extname = txt
|
||||||
|
txt-encoding = UTF-8
|
||||||
|
preserve-chapter-cache = 0
|
||||||
|
# 工具模式不需要终端进度条，日志统一走 stdout 捕获
|
||||||
|
enable-progressbar = 0
|
||||||
|
|
||||||
|
[source]
|
||||||
|
language = zh-CN
|
||||||
|
# 书源规则绝对路径，避免依赖运行目录
|
||||||
|
active-rules = @RULES@
|
||||||
|
source-id =
|
||||||
|
search-limit = 30
|
||||||
|
search-filter = 1
|
||||||
|
|
||||||
|
[crawl]
|
||||||
|
concurrency =
|
||||||
|
min-interval = 200
|
||||||
|
max-interval = 400
|
||||||
|
enable-retry = 1
|
||||||
|
max-retries = 3
|
||||||
|
retry-min-interval = 2000
|
||||||
|
retry-max-interval = 4000
|
||||||
|
|
||||||
|
[web]
|
||||||
|
enabled = 0
|
||||||
|
port = 7765
|
||||||
|
|
||||||
|
[cookie]
|
||||||
|
qidian =
|
||||||
|
|
||||||
|
[proxy]
|
||||||
|
enabled = 0
|
||||||
|
host = 127.0.0.1
|
||||||
|
port = 7890
|
||||||
+127
@@ -0,0 +1,127 @@
|
|||||||
|
# Shared helpers for the ebook-skill command line (Windows PowerShell 5.1+ / pwsh 7+).
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
|
||||||
|
function Get-ProjectRoot {
|
||||||
|
param([string]$Start = $PSScriptRoot)
|
||||||
|
$dir = (Resolve-Path $Start).Path
|
||||||
|
while ($dir) {
|
||||||
|
if (Test-Path (Join-Path $dir 'vendor\so-novel\pom.xml')) { return $dir }
|
||||||
|
$parent = Split-Path $dir -Parent
|
||||||
|
if ($parent -eq $dir -or -not $parent) { break }
|
||||||
|
$dir = $parent
|
||||||
|
}
|
||||||
|
throw "Cannot locate project root (vendor/so-novel/pom.xml not found above $Start). Run: git submodule update --init"
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-ToolPaths {
|
||||||
|
param([string]$Root = (Get-ProjectRoot))
|
||||||
|
[pscustomobject]@{
|
||||||
|
Root = $Root
|
||||||
|
Vendor = Join-Path $Root 'vendor\so-novel'
|
||||||
|
Tools = Join-Path $Root 'tools'
|
||||||
|
Runtime = Join-Path $Root 'tools\runtime'
|
||||||
|
Build = Join-Path $Root 'tools\build'
|
||||||
|
Classes = Join-Path $Root 'tools\build\classes'
|
||||||
|
AppJar = Join-Path $Root 'tools\runtime\app.jar'
|
||||||
|
ToolJar = Join-Path $Root 'tools\runtime\sonovel-tool.jar'
|
||||||
|
Config = Join-Path $Root 'tools\runtime\config.ini'
|
||||||
|
Rules = Join-Path $Root 'tools\runtime\rules'
|
||||||
|
Books = Join-Path $Root 'books'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Test-Jdk21 {
|
||||||
|
param([string]$JdkHome)
|
||||||
|
if (-not $JdkHome) { return $false }
|
||||||
|
$java = Join-Path $JdkHome 'bin\java.exe'
|
||||||
|
if (-not (Test-Path $java)) { return $false }
|
||||||
|
try {
|
||||||
|
$out = (& $java -version 2>&1 | Out-String)
|
||||||
|
} catch {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
if ($out -match 'version "(?:1\.)?(\d+)') {
|
||||||
|
return ([int]$Matches[1] -ge 21)
|
||||||
|
}
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
|
||||||
|
function Find-Jdk21 {
|
||||||
|
param([switch]$AllowDownload)
|
||||||
|
|
||||||
|
$candidates = New-Object System.Collections.Generic.List[string]
|
||||||
|
if ($env:EBOOK_NOVEL_JAVA_HOME) { $candidates.Add($env:EBOOK_NOVEL_JAVA_HOME) }
|
||||||
|
if ($env:JAVA_HOME) { $candidates.Add($env:JAVA_HOME) }
|
||||||
|
|
||||||
|
$patterns = @(
|
||||||
|
(Join-Path $PSScriptRoot '.jdk\*\*'),
|
||||||
|
(Join-Path $PSScriptRoot '.jdk'),
|
||||||
|
'C:\Program Files\Java\jdk-21*',
|
||||||
|
'C:\Program Files\Eclipse Adoptium\jdk-21*',
|
||||||
|
'C:\Program Files\JetBrains\*\jbr',
|
||||||
|
(Join-Path $env:LOCALAPPDATA 'mise\installs\java\21*'),
|
||||||
|
(Join-Path $env:LOCALAPPDATA 'mise\installs\java\2*')
|
||||||
|
)
|
||||||
|
foreach ($pattern in $patterns) {
|
||||||
|
foreach ($dir in (Get-ChildItem -Path $pattern -Directory -ErrorAction SilentlyContinue)) {
|
||||||
|
$candidates.Add($dir.FullName)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($candidate in $candidates) {
|
||||||
|
if (Test-Jdk21 -JdkHome $candidate) { return (Resolve-Path $candidate).Path }
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($AllowDownload) { return (Install-Jdk21) }
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
|
# Downloads a portable Temurin JDK 21 (used only when no local JDK 21 exists).
|
||||||
|
function Install-Jdk21 {
|
||||||
|
$dest = Join-Path $PSScriptRoot '.jdk'
|
||||||
|
New-Item -ItemType Directory -Force $dest | Out-Null
|
||||||
|
|
||||||
|
Write-Host '[setup] no JDK 21 found, downloading Temurin 21 from the Tsinghua mirror ...' -ForegroundColor Yellow
|
||||||
|
$mirror = 'https://mirrors.tuna.tsinghua.edu.cn/Adoptium/21/jdk/x64/windows/'
|
||||||
|
$listing = (Invoke-WebRequest -Uri $mirror -UseBasicParsing -TimeoutSec 60).Content
|
||||||
|
$names = [regex]::Matches($listing, 'OpenJDK21U-jdk_x64_windows_hotspot_[0-9._]+\.zip') |
|
||||||
|
ForEach-Object { $_.Value } | Sort-Object -Unique
|
||||||
|
if (-not $names) { throw "Could not find a JDK 21 archive at $mirror" }
|
||||||
|
$zipName = $names[$names.Count - 1]
|
||||||
|
|
||||||
|
$zip = Join-Path $env:TEMP $zipName
|
||||||
|
if (-not (Test-Path $zip)) {
|
||||||
|
Invoke-WebRequest -Uri ($mirror + $zipName) -OutFile $zip -UseBasicParsing -TimeoutSec 1800
|
||||||
|
}
|
||||||
|
Expand-Archive -Path $zip -DestinationPath $dest -Force
|
||||||
|
|
||||||
|
$java = Get-ChildItem -Path (Join-Path $dest '*\bin\java.exe') -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||||
|
if (-not $java) { throw "JDK extraction failed under $dest" }
|
||||||
|
$JdkHome = Split-Path (Split-Path $java.FullName -Parent) -Parent
|
||||||
|
Write-Host "[setup] JDK 21 installed at $JdkHome" -ForegroundColor Green
|
||||||
|
return $JdkHome
|
||||||
|
}
|
||||||
|
|
||||||
|
function Get-JavaCommand {
|
||||||
|
param([string]$Root = (Get-ProjectRoot))
|
||||||
|
$jdkHome = Find-Jdk21
|
||||||
|
if (-not $jdkHome) {
|
||||||
|
throw 'JDK 21 is required. Run tools\build.ps1 (it can download one), or set EBOOK_NOVEL_JAVA_HOME.'
|
||||||
|
}
|
||||||
|
return (Join-Path $jdkHome 'bin\java.exe')
|
||||||
|
}
|
||||||
|
|
||||||
|
function Find-JarTool {
|
||||||
|
# JBR ships without jar.exe, so fall back to any other JDK on the machine.
|
||||||
|
foreach ($pattern in @('C:\Program Files\Java\*\bin\jar.exe',
|
||||||
|
(Join-Path $env:LOCALAPPDATA 'mise\installs\java\*\bin\jar.exe'),
|
||||||
|
'C:\Program Files\Eclipse Adoptium\*\bin\jar.exe')) {
|
||||||
|
$exe = Get-ChildItem -Path $pattern -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||||
|
if ($exe) { return $exe.FullName }
|
||||||
|
}
|
||||||
|
$cmd = Get-Command jar.exe -ErrorAction SilentlyContinue
|
||||||
|
if ($cmd) { return $cmd.Source }
|
||||||
|
return $null
|
||||||
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,482 @@
|
|||||||
|
package skills.novel;
|
||||||
|
|
||||||
|
import cn.hutool.core.util.StrUtil;
|
||||||
|
import cn.hutool.json.JSONArray;
|
||||||
|
import cn.hutool.json.JSONObject;
|
||||||
|
import cn.hutool.json.JSONUtil;
|
||||||
|
import com.pcdd.sonovel.context.HttpClientContext;
|
||||||
|
import com.pcdd.sonovel.context.OkHttpClientFactory;
|
||||||
|
import com.pcdd.sonovel.core.AppConfigLoader;
|
||||||
|
import com.pcdd.sonovel.core.Crawler;
|
||||||
|
import com.pcdd.sonovel.core.Source;
|
||||||
|
import com.pcdd.sonovel.handler.SearchResultsHandler;
|
||||||
|
import com.pcdd.sonovel.model.AppConfig;
|
||||||
|
import com.pcdd.sonovel.model.Chapter;
|
||||||
|
import com.pcdd.sonovel.model.Rule;
|
||||||
|
import com.pcdd.sonovel.model.Rule.Book;
|
||||||
|
import com.pcdd.sonovel.model.SearchResult;
|
||||||
|
import com.pcdd.sonovel.parser.BookParser;
|
||||||
|
import com.pcdd.sonovel.parser.SearchParser;
|
||||||
|
import com.pcdd.sonovel.parser.TocParser;
|
||||||
|
import com.pcdd.sonovel.utils.SourceUtils;
|
||||||
|
|
||||||
|
import java.io.ByteArrayOutputStream;
|
||||||
|
import java.io.File;
|
||||||
|
import java.io.FileDescriptor;
|
||||||
|
import java.io.FileOutputStream;
|
||||||
|
import java.io.OutputStream;
|
||||||
|
import java.io.PrintStream;
|
||||||
|
import java.nio.charset.StandardCharsets;
|
||||||
|
import java.util.ArrayList;
|
||||||
|
import java.util.Arrays;
|
||||||
|
import java.util.Comparator;
|
||||||
|
import java.util.List;
|
||||||
|
import java.util.Map;
|
||||||
|
import java.util.concurrent.ConcurrentLinkedQueue;
|
||||||
|
import java.util.concurrent.CountDownLatch;
|
||||||
|
import java.util.concurrent.Executors;
|
||||||
|
import java.util.concurrent.TimeUnit;
|
||||||
|
import java.util.concurrent.atomic.AtomicInteger;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Non-interactive wrapper around so-novel's crawler library.
|
||||||
|
*
|
||||||
|
* <p>so-novel ships as TUI / Web / single-URL CLI. This entry point adds the missing
|
||||||
|
* piece for tool usage: aggregated keyword search with machine-readable output, plus a
|
||||||
|
* download command that reports the produced file. Everything else (rule engine,
|
||||||
|
* HTTP layer, chapter concurrency, epub/txt merging) is delegated to so-novel itself.
|
||||||
|
*
|
||||||
|
* <p>stdout carries exactly one JSON document; all library console output is captured
|
||||||
|
* and returned in {@code logTail} when {@code --log} is given.
|
||||||
|
*/
|
||||||
|
public final class ToolMain {
|
||||||
|
|
||||||
|
private static final PrintStream OUT =
|
||||||
|
new PrintStream(new FileOutputStream(FileDescriptor.out), true, StandardCharsets.UTF_8);
|
||||||
|
|
||||||
|
private static final int LOG_TAIL_BYTES = 64 * 1024;
|
||||||
|
|
||||||
|
public static void main(String[] args) {
|
||||||
|
TailStream tail = new TailStream(LOG_TAIL_BYTES);
|
||||||
|
PrintStream sink = new PrintStream(tail, true, StandardCharsets.UTF_8);
|
||||||
|
PrintStream originalOut = System.out;
|
||||||
|
System.setOut(sink);
|
||||||
|
System.setErr(sink);
|
||||||
|
|
||||||
|
JSONObject result;
|
||||||
|
try {
|
||||||
|
result = dispatch(args);
|
||||||
|
} catch (Throwable t) {
|
||||||
|
result = new JSONObject();
|
||||||
|
result.set("ok", false);
|
||||||
|
result.set("error", t.getClass().getSimpleName() + ": " + String.valueOf(t.getMessage()));
|
||||||
|
result.set("detail", StrUtil.sub(stackTrace(t), 0, 2000));
|
||||||
|
} finally {
|
||||||
|
System.setOut(originalOut);
|
||||||
|
System.setErr(originalOut);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (hasFlag(args, "--log")) {
|
||||||
|
result.set("logTail", tail.tailLines(40));
|
||||||
|
}
|
||||||
|
OUT.println(escapeNonAscii(JSONUtil.toJsonPrettyStr(result)));
|
||||||
|
OUT.flush();
|
||||||
|
|
||||||
|
// so-novel opens pooled V8 runtimes and okhttp dispatchers; exit explicitly.
|
||||||
|
System.exit(Boolean.TRUE.equals(result.getBool("ok")) ? 0 : 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static JSONObject dispatch(String[] args) {
|
||||||
|
if (args.length == 0 || hasFlag(args, "-h") || hasFlag(args, "--help")) {
|
||||||
|
JSONObject o = new JSONObject();
|
||||||
|
o.set("ok", false);
|
||||||
|
o.set("error", "missing command");
|
||||||
|
o.set("usage", new JSONArray(List.of(
|
||||||
|
"search <keyword> [--limit N] [--source ID] [--timeout SEC] [--log]",
|
||||||
|
"download <detail-url> [--ext txt|epub|html|pdf] [--out DIR] [--log]",
|
||||||
|
"toc <detail-url> [--log]",
|
||||||
|
"sources")));
|
||||||
|
return o;
|
||||||
|
}
|
||||||
|
|
||||||
|
String command = args[0];
|
||||||
|
return switch (command) {
|
||||||
|
case "search" -> search(positional(args, 1), intOpt(args, "--limit", -1),
|
||||||
|
intOpt(args, "--source", -1), intOpt(args, "--timeout", 180));
|
||||||
|
case "download" -> download(positional(args, 1).isEmpty() ? opt(args, "--url") : positional(args, 1).get(0),
|
||||||
|
opt(args, "--ext", "txt"), opt(args, "--out", null));
|
||||||
|
case "toc" -> toc(positional(args, 1).isEmpty() ? opt(args, "--url") : positional(args, 1).get(0),
|
||||||
|
intOpt(args, "--limit", 10));
|
||||||
|
case "sources" -> sources();
|
||||||
|
default -> {
|
||||||
|
JSONObject o = new JSONObject();
|
||||||
|
o.set("ok", false);
|
||||||
|
o.set("error", "unknown command: " + command);
|
||||||
|
yield o;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------- commands
|
||||||
|
|
||||||
|
private static JSONObject search(List<String> words, int limit, int sourceId, int timeoutSec) {
|
||||||
|
String keyword = String.join(" ", words).trim();
|
||||||
|
JSONObject out = new JSONObject();
|
||||||
|
out.set("ok", false);
|
||||||
|
out.set("command", "search");
|
||||||
|
if (keyword.isEmpty()) {
|
||||||
|
out.set("error", "keyword is required");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
out.set("keyword", keyword);
|
||||||
|
|
||||||
|
AppConfig cfg = AppConfigLoader.APP_CONFIG;
|
||||||
|
cfg.setEnableProgressbar(0);
|
||||||
|
cfg.setAutoUpdate(0);
|
||||||
|
if (limit > 0) cfg.setSearchLimit(limit);
|
||||||
|
HttpClientContext.set(OkHttpClientFactory.create(cfg));
|
||||||
|
warmUpJsRuntime(cfg);
|
||||||
|
|
||||||
|
long start = System.currentTimeMillis();
|
||||||
|
List<Source> sources = new ArrayList<>(SourceUtils.getSearchableSources());
|
||||||
|
if (sourceId > 0) {
|
||||||
|
sources.removeIf(s -> s.rule.getId() != sourceId);
|
||||||
|
}
|
||||||
|
|
||||||
|
ConcurrentLinkedQueue<SearchResult> collected = new ConcurrentLinkedQueue<>();
|
||||||
|
ConcurrentLinkedQueue<JSONObject> failed = new ConcurrentLinkedQueue<>();
|
||||||
|
AtomicInteger empty = new AtomicInteger();
|
||||||
|
CountDownLatch latch = new CountDownLatch(sources.size());
|
||||||
|
|
||||||
|
try (var pool = Executors.newVirtualThreadPerTaskExecutor()) {
|
||||||
|
for (Source source : sources) {
|
||||||
|
int effectiveLimit = limit > 0 ? limit : cfg.getSearchLimit();
|
||||||
|
if (effectiveLimit > 0) source.config.setSearchLimit(effectiveLimit);
|
||||||
|
source.config.setEnableProgressbar(0);
|
||||||
|
|
||||||
|
pool.execute(() -> {
|
||||||
|
long t0 = System.currentTimeMillis();
|
||||||
|
try {
|
||||||
|
List<SearchResult> hits = new SearchParser(source.config).parse(keyword);
|
||||||
|
if (hits == null || hits.isEmpty()) {
|
||||||
|
empty.incrementAndGet();
|
||||||
|
} else {
|
||||||
|
collected.addAll(hits);
|
||||||
|
}
|
||||||
|
} catch (Throwable e) {
|
||||||
|
JSONObject f = new JSONObject();
|
||||||
|
f.set("sourceId", source.rule.getId());
|
||||||
|
f.set("sourceName", source.rule.getName());
|
||||||
|
f.set("error", String.valueOf(e.getMessage()));
|
||||||
|
f.set("ms", System.currentTimeMillis() - t0);
|
||||||
|
failed.add(f);
|
||||||
|
} finally {
|
||||||
|
latch.countDown();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
if (!latch.await(timeoutSec, TimeUnit.SECONDS)) {
|
||||||
|
out.set("timedOut", true);
|
||||||
|
}
|
||||||
|
} catch (InterruptedException e) {
|
||||||
|
Thread.currentThread().interrupt();
|
||||||
|
out.set("error", "interrupted: " + e.getMessage());
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
List<SearchResult> ranked = SearchResultsHandler.filterAndSort(new ArrayList<>(collected), keyword);
|
||||||
|
|
||||||
|
JSONArray rows = new JSONArray();
|
||||||
|
int index = 1;
|
||||||
|
for (SearchResult sr : ranked) {
|
||||||
|
JSONObject row = new JSONObject();
|
||||||
|
row.set("index", index++);
|
||||||
|
row.set("sourceId", sr.getSourceId());
|
||||||
|
row.set("sourceName", sr.getSourceName());
|
||||||
|
row.set("bookName", sr.getBookName());
|
||||||
|
row.set("author", sr.getAuthor());
|
||||||
|
row.set("latestChapter", sr.getLatestChapter());
|
||||||
|
row.set("lastUpdateTime", sr.getLastUpdateTime());
|
||||||
|
row.set("status", sr.getStatus());
|
||||||
|
row.set("category", sr.getCategory());
|
||||||
|
row.set("wordCount", sr.getWordCount());
|
||||||
|
row.set("url", sr.getUrl());
|
||||||
|
rows.add(row);
|
||||||
|
}
|
||||||
|
|
||||||
|
JSONObject stats = new JSONObject();
|
||||||
|
stats.set("total", sources.size());
|
||||||
|
stats.set("returned", collected.size());
|
||||||
|
stats.set("empty", empty.get());
|
||||||
|
stats.set("failed", new JSONArray(failed));
|
||||||
|
|
||||||
|
out.set("ok", true);
|
||||||
|
out.set("count", rows.size());
|
||||||
|
out.set("results", rows);
|
||||||
|
out.set("sources", stats);
|
||||||
|
out.set("tookMs", System.currentTimeMillis() - start);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static JSONObject download(String url, String ext, String outDir) {
|
||||||
|
JSONObject out = new JSONObject();
|
||||||
|
out.set("ok", false);
|
||||||
|
out.set("command", "download");
|
||||||
|
if (StrUtil.isBlank(url)) {
|
||||||
|
out.set("error", "book detail url is required (--url or first positional argument)");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
String format = ext == null ? "txt" : ext.toLowerCase();
|
||||||
|
if (!format.matches("txt|epub|html|pdf")) {
|
||||||
|
out.set("error", "unsupported ext: " + ext + " (txt|epub|html|pdf)");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
out.set("url", url);
|
||||||
|
out.set("ext", format);
|
||||||
|
|
||||||
|
AppConfig cfg = AppConfigLoader.APP_CONFIG;
|
||||||
|
cfg.setEnableProgressbar(0);
|
||||||
|
cfg.setAutoUpdate(0);
|
||||||
|
cfg.setExtName(format);
|
||||||
|
if (StrUtil.isNotBlank(outDir)) cfg.setDownloadPath(new File(outDir).getAbsolutePath());
|
||||||
|
|
||||||
|
// bookUrl -> rule: same lookup the TUI/CLI uses, so any active source is accepted
|
||||||
|
Rule rule = SourceUtils.getRule(url);
|
||||||
|
cfg.setSourceId(rule.getId());
|
||||||
|
out.set("sourceId", rule.getId());
|
||||||
|
out.set("sourceName", rule.getName());
|
||||||
|
|
||||||
|
HttpClientContext.set(OkHttpClientFactory.create(cfg));
|
||||||
|
warmUpJsRuntime(cfg);
|
||||||
|
|
||||||
|
Book book = new BookParser(cfg).parse(url);
|
||||||
|
out.set("bookName", book.getBookName());
|
||||||
|
out.set("author", book.getAuthor());
|
||||||
|
|
||||||
|
List<Chapter> toc = new TocParser(cfg).parseAll(url);
|
||||||
|
out.set("chapters", toc.size());
|
||||||
|
if (toc.isEmpty()) {
|
||||||
|
out.set("error", "source returned an empty table of contents (rule broken or page blocked)");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
long start = System.currentTimeMillis();
|
||||||
|
// reuse the parsed toc: avoids a second directory fetch and lets us report the chapter count
|
||||||
|
double seconds = new Crawler(cfg).crawl(url, toc);
|
||||||
|
long tookMs = System.currentTimeMillis() - start;
|
||||||
|
|
||||||
|
File produced = locateOutput(cfg.getDownloadPath(), book, format);
|
||||||
|
boolean ok = produced != null && produced.length() > 0;
|
||||||
|
out.set("ok", ok);
|
||||||
|
out.set("seconds", Math.round(seconds * 100.0) / 100.0);
|
||||||
|
out.set("tookMs", tookMs);
|
||||||
|
out.set("downloadPath", new File(cfg.getDownloadPath()).getAbsolutePath());
|
||||||
|
if (ok) {
|
||||||
|
out.set("file", produced.getAbsolutePath());
|
||||||
|
out.set("bytes", produced.length());
|
||||||
|
} else {
|
||||||
|
out.set("error", "download finished but no output file was produced "
|
||||||
|
+ "(source probably returned an empty table of contents)");
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static JSONObject toc(String url, int preview) {
|
||||||
|
JSONObject out = new JSONObject();
|
||||||
|
out.set("ok", false);
|
||||||
|
out.set("command", "toc");
|
||||||
|
if (StrUtil.isBlank(url)) {
|
||||||
|
out.set("error", "book detail url is required");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
AppConfig cfg = AppConfigLoader.APP_CONFIG;
|
||||||
|
cfg.setEnableProgressbar(0);
|
||||||
|
cfg.setAutoUpdate(0);
|
||||||
|
Rule rule = SourceUtils.getRule(url);
|
||||||
|
cfg.setSourceId(rule.getId());
|
||||||
|
|
||||||
|
HttpClientContext.set(OkHttpClientFactory.create(cfg));
|
||||||
|
warmUpJsRuntime(cfg);
|
||||||
|
|
||||||
|
List<Chapter> chapters = new TocParser(cfg).parseAll(url);
|
||||||
|
JSONArray head = new JSONArray();
|
||||||
|
JSONArray tail = new JSONArray();
|
||||||
|
for (int i = 0; i < chapters.size(); i++) {
|
||||||
|
if (i < preview) head.add(chapters.get(i).getTitle());
|
||||||
|
if (i >= chapters.size() - preview) tail.add(chapters.get(i).getTitle());
|
||||||
|
}
|
||||||
|
out.set("ok", !chapters.isEmpty());
|
||||||
|
out.set("url", url);
|
||||||
|
out.set("sourceId", rule.getId());
|
||||||
|
out.set("sourceName", rule.getName());
|
||||||
|
out.set("chapters", chapters.size());
|
||||||
|
out.set("firstChapters", head);
|
||||||
|
out.set("lastChapters", tail);
|
||||||
|
if (chapters.isEmpty()) out.set("error", "empty table of contents");
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static JSONObject sources() {
|
||||||
|
AppConfig cfg = AppConfigLoader.APP_CONFIG;
|
||||||
|
cfg.setAutoUpdate(0);
|
||||||
|
JSONArray rows = new JSONArray();
|
||||||
|
for (Rule r : SourceUtils.getActivatedRules()) {
|
||||||
|
JSONObject row = new JSONObject();
|
||||||
|
row.set("id", r.getId());
|
||||||
|
row.set("name", r.getName());
|
||||||
|
row.set("url", r.getUrl());
|
||||||
|
row.set("searchable", r.getSearch() != null && !r.getSearch().isDisabled() && !r.isDisabled());
|
||||||
|
row.set("language", r.getLanguage());
|
||||||
|
row.set("needProxy", r.isNeedProxy());
|
||||||
|
rows.add(row);
|
||||||
|
}
|
||||||
|
JSONObject out = new JSONObject();
|
||||||
|
out.set("ok", true);
|
||||||
|
out.set("command", "sources");
|
||||||
|
out.set("activeRules", cfg.getActiveRules());
|
||||||
|
out.set("count", rows.size());
|
||||||
|
out.set("sources", rows);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------- helpers
|
||||||
|
|
||||||
|
/** Creates and closes one V8 runtime, mirroring com.pcdd.sonovel.Main's Windows workaround. */
|
||||||
|
private static void warmUpJsRuntime(AppConfig cfg) {
|
||||||
|
try (var v8 = com.caoccao.javet.interop.V8Host.getV8Instance().createV8Runtime()) {
|
||||||
|
// no-op
|
||||||
|
} catch (Throwable ignored) {
|
||||||
|
// JS-based rules are optional; CSS/XPath rules still work without V8
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static File locateOutput(String downloadPath, Book book, String ext) {
|
||||||
|
File dir = new File(downloadPath);
|
||||||
|
File direct = new File(dir, "%s(%s).%s".formatted(book.getBookName(), book.getAuthor(), ext));
|
||||||
|
if (direct.exists()) return direct;
|
||||||
|
|
||||||
|
File[] candidates = dir.listFiles((d, name) -> name.toLowerCase().endsWith("." + ext));
|
||||||
|
if (candidates == null || candidates.length == 0) return null;
|
||||||
|
return Arrays.stream(candidates)
|
||||||
|
.filter(f -> f.getName().contains(book.getBookName()))
|
||||||
|
.max(Comparator.comparingLong(File::lastModified))
|
||||||
|
.orElseGet(() -> Arrays.stream(candidates)
|
||||||
|
.max(Comparator.comparingLong(File::lastModified))
|
||||||
|
.orElse(null));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static List<String> positional(String[] args, int from) {
|
||||||
|
List<String> values = new ArrayList<>();
|
||||||
|
for (int i = from; i < args.length; i++) {
|
||||||
|
String a = args[i];
|
||||||
|
if (a.startsWith("--")) {
|
||||||
|
// skip option values that are not positional
|
||||||
|
if (needsValue(a) && i + 1 < args.length) i++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
values.add(a);
|
||||||
|
}
|
||||||
|
return values;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String opt(String[] args, String name, String fallback) {
|
||||||
|
String value = opt(args, name);
|
||||||
|
return value == null ? fallback : value;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String opt(String[] args, String name) {
|
||||||
|
for (int i = 0; i < args.length; i++) {
|
||||||
|
String a = args[i];
|
||||||
|
if (a.equals(name) && i + 1 < args.length) return args[i + 1];
|
||||||
|
if (a.startsWith(name + "=")) return a.substring(name.length() + 1);
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int intOpt(String[] args, String name, int fallback) {
|
||||||
|
String v = opt(args, name);
|
||||||
|
if (StrUtil.isBlank(v)) return fallback;
|
||||||
|
try {
|
||||||
|
return Integer.parseInt(v.trim());
|
||||||
|
} catch (NumberFormatException e) {
|
||||||
|
return fallback;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static boolean hasFlag(String[] args, String name) {
|
||||||
|
return Arrays.stream(args).anyMatch(a -> a.equalsIgnoreCase(name));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static boolean needsValue(String option) {
|
||||||
|
String bare = option.contains("=") ? option.substring(0, option.indexOf('=')) : option;
|
||||||
|
return switch (bare) {
|
||||||
|
case "--limit", "--source", "--timeout", "--ext", "--out", "--url" -> true;
|
||||||
|
default -> false;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static String stackTrace(Throwable t) {
|
||||||
|
java.io.StringWriter sw = new java.io.StringWriter();
|
||||||
|
t.printStackTrace(new java.io.PrintWriter(sw));
|
||||||
|
return sw.toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* JSON only needs ASCII: escaping non-ASCII keeps the output byte-identical under any
|
||||||
|
* console code page, which avoids mojibake in Windows shells and pipelines.
|
||||||
|
*/
|
||||||
|
private static String escapeNonAscii(String json) {
|
||||||
|
StringBuilder sb = new StringBuilder(json.length() + 64);
|
||||||
|
for (int i = 0; i < json.length(); i++) {
|
||||||
|
char c = json.charAt(i);
|
||||||
|
if (c < 0x80) {
|
||||||
|
sb.append(c);
|
||||||
|
} else {
|
||||||
|
sb.append(String.format("\\u%04x", (int) c));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sb.toString();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Bounded byte buffer that keeps the last N bytes written by the library. */
|
||||||
|
private static final class TailStream extends OutputStream {
|
||||||
|
private final int max;
|
||||||
|
private final ByteArrayOutputStream buffer = new ByteArrayOutputStream();
|
||||||
|
|
||||||
|
TailStream(int max) {
|
||||||
|
this.max = max;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized void write(int b) {
|
||||||
|
trimIfNeeded(1);
|
||||||
|
buffer.write(b);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public synchronized void write(byte[] b, int off, int len) {
|
||||||
|
trimIfNeeded(len);
|
||||||
|
buffer.write(b, off, len);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void trimIfNeeded(int incoming) {
|
||||||
|
if (buffer.size() + incoming <= max) return;
|
||||||
|
byte[] keep = Arrays.copyOfRange(buffer.toByteArray(), max / 2, buffer.size());
|
||||||
|
buffer.reset();
|
||||||
|
buffer.write(keep, 0, keep.length);
|
||||||
|
}
|
||||||
|
|
||||||
|
synchronized List<String> tailLines(int count) {
|
||||||
|
String text = buffer.toString(StandardCharsets.UTF_8);
|
||||||
|
List<String> lines = Arrays.stream(text.split("\\R"))
|
||||||
|
.filter(StrUtil::isNotBlank)
|
||||||
|
.toList();
|
||||||
|
int from = Math.max(0, lines.size() - count);
|
||||||
|
return new ArrayList<>(lines.subList(from, lines.size()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private ToolMain() {
|
||||||
|
}
|
||||||
|
}
|
||||||
+1
Submodule vendor/so-novel added at 88acec6ee4
Reference in New Issue
Block a user