78 Commits

Author SHA1 Message Date
Godzil
cc0bf4dfb1 1.5.0 2019-07-31 17:20:53 +02:00
Godzil
00857ba46f Update dependencies 2019-07-31 17:20:47 +02:00
Godzil
b77a35e0e9 Remove non production logs 2019-07-31 17:16:30 +02:00
Manoël Trapier
ca59e3b2fd Merge pull request #108 from Ronserruya/fix_title_scrape
Fix login and only direct links issues.

Still need to understand what is happening.
2019-07-31 16:09:27 +01:00
ronserruya
95c0c4d6d3 Linter stuff 2019-07-31 14:29:49 +03:00
ronserruya
0d2d36251a Fix title fetching 2019-07-31 14:26:22 +03:00
ronserruya
48a58ffca6 Fix login issue 2019-07-31 14:26:13 +03:00
Manoël Trapier
505e6c67ce Add one more badge 2019-05-07 13:39:04 +01:00
Manoël Trapier
83e8a5e08c Change issue badge 2019-05-07 13:28:01 +01:00
Godzil
c82319a2c6 Make code (somwhat) compliant with latest version of CloudScraper and add some instrumentation to try to understand what is happening with web based login. Still unclear for now.. 2019-05-07 13:13:08 +02:00
Godzil
1fe7c697c5 Remove the old and unneeded ts.js 2019-05-07 13:11:03 +02:00
Godzil
239d1c60a3 Update some dependencies 2019-05-07 13:10:34 +02:00
Godzil
bdfc96d56e Remove package-lock.json as it is not needed 2019-04-30 16:57:25 +02:00
Godzil
8f7babd809 1.4.6 2019-03-04 18:51:27 +01:00
Godzil
c708df574b Update deps 2019-03-04 18:49:29 +01:00
Godzil
401a511668 Add Node 10 and 11 2019-03-04 18:44:41 +01:00
Godzil
969879921e 1.4.5 2018-10-04 20:08:15 +01:00
Godzil
546ba9b45a Add a warn when login failed to be more explicit 2018-10-04 20:07:54 +01:00
Godzil
27bdf54782 Solve issue with redirection (now it should follow automatically) 2018-10-04 20:02:28 +01:00
Godzil
beed932e93 Javascript: I hate you.
(fix a **** stupid bug while doing version checking)
2018-08-27 18:16:23 +01:00
Godzil
e5c4c08e66 1.4.4 2018-08-27 16:46:59 +01:00
Godzil
2b201b0785 Fix #94 2018-08-27 13:16:22 +01:00
Godzil
fdf5805911 Fix for #88 2018-08-27 13:11:06 +01:00
Godzil
9191075f48 Fix for #92 when the version server is not answering properly 2018-08-27 13:08:01 +01:00
Godzil
9f73e4f865 Update `ignoredub` to support more form
(and also make it work with multiple languages)
2018-08-17 00:56:50 +01:00
Manoël Trapier
1f20e028e1 Merge pull request #87 from TheDammedGamer/master
Filtering out Pipe Symbol in file names
2018-08-13 15:36:35 +01:00
Liam Townsend
da0fb17015 Filtering out Pipe Symbol in file names
updated sanitiseFileName to include the pipe symbol as this character is not allowed on windows, the error thast it currently throws is: {"errno":-4058,"code":"ENOENT","syscall":"open","path":"K:\\MediaDwn\\Is It Wrong to Try to Pick Up Girls in a Dungeon_\\Is It Wrong to Try to Pick Up Girls in a Dungeon_ - s01e01 - Bell Cranel | Adventurer - [CrunchyRoll].ass"}
2018-08-13 15:25:06 +01:00
Godzil
2aa71832b3 1.4.3 2018-08-11 20:50:02 +01:00
Godzil
876def4392 Add code to check what langage CR is serving the page, and try to adapt
some regexp to that. The langage can be forced by the user

Fix #1 and Fix #76
2018-08-11 20:42:12 +01:00
Godzil
0ba51b7270 1.4.2 2018-08-05 11:01:03 +01:00
Godzil
7da4289097 1.4.2-0 2018-08-05 10:57:09 +01:00
Godzil
ce5038cf08 @ URL was broken since 1.3.7 doh! 2018-08-05 10:46:27 +01:00
Godzil
1b0f53a88c 1.4.1 2018-08-05 10:34:00 +01:00
Godzil
d19992f0d3 Make linter happy (and fix a mistake) 2018-08-05 10:30:22 +01:00
Godzil
a44d1ae668 Use a more stable and futur proof URL to get current version information 2018-08-05 09:30:14 +01:00
Godzil
14fd18479e Try to tweak ffmpeg setting, but there are still some stalling issues.. 2018-08-05 09:29:39 +01:00
Godzil
1106a28288 Make possible georestrictions messages more clear 2018-08-05 09:28:53 +01:00
Godzil
f1a5d1b6a8 Add a new warning type (more "strong" but still a warning) 2018-08-05 09:28:28 +01:00
Godzil
4193643306 Fix a missing invalid char for filename for windows ('\') 2018-08-05 09:27:46 +01:00
Godzil
ebe671ff5b 1.4.0 2018-08-01 23:20:49 +01:00
Godzil
fd447f2cc6 Update readme with some information about config file 2018-08-01 23:20:02 +01:00
Godzil
7dcd932ee5 Add a warning message is a license infobox is display to say that maybe
some episodes would be missing

COOOOKIIIEEEEE!!!!

(I think I'm hungry, I should go to eat!)
2018-08-01 22:40:45 +01:00
Godzil
ed233de565 Add an experimental feature: ignoring episodes from season that end with 'dub)' as they are dubbed seasons. 2018-08-01 22:08:21 +01:00
Godzil
a679573bf3 Add support to change the user agent. 2018-08-01 21:26:11 +01:00
Godzil
24d6892261 Linter, I hate you with your stupid remarks. 2018-08-01 21:22:28 +01:00
Godzil
25dabd4955 Fix #80, the batch file path should not be related to the output folder.
Also make sure that an absolute path is not treated as a relative one!



COOOKIESSSS!!!!!!
2018-08-01 21:21:46 +01:00
Godzil
ce65324c57 Add a message before login just to look pretty. 2018-08-01 20:38:14 +01:00
Godzil
a0f10252a1 Add back the old login method as fallback. 2018-08-01 20:37:50 +01:00
Godzil
6e638488dc Update user agent. 2018-08-01 20:37:07 +01:00
Godzil
2e8de8c5c2 Make lint happy (sorry cookie monster, nothing for you here :( ) 2018-08-01 19:57:29 +01:00
Godzil
9c3aaf220a Make authentification error report to work, and
warn user if trying to use API but not filling corresponding fields
2018-08-01 19:56:49 +01:00
Godzil
ab35bb4439 Add @ssttevee method of authentication (from pull request #43) 2018-08-01 02:07:21 +01:00
Godzil
b48877b786 Prepare to add multiples logins methods 2018-08-01 02:05:15 +01:00
Godzil
9fb85d4376 Now that we have cookies, persistant config file, we can log
and stay logged between run (or I hope so) so add a delog command,
and make the cookie monster happy.
2018-08-01 00:52:23 +01:00
Godzil
a582b15103 Check if the we got proper information about the session,
else die with a lot of suffers!
2018-08-01 00:50:38 +01:00
Godzil
da3a51991c Properly use the config info for API endpoint informations. 2018-08-01 00:49:49 +01:00
Godzil
22f70c86f5 Add a function to make the cookie monster happy! 2018-08-01 00:48:38 +01:00
Godzil
0daf4d895f Stop silently ignore login issues 2018-08-01 00:47:57 +01:00
Godzil
80165a76e0 Use a proper UUID v4 instead of the weird code to generate a device id 2018-08-01 00:47:33 +01:00
Godzil
a6b025bdbf Add a proper cookie store. 2018-08-01 00:46:14 +01:00
Godzil
02a9ed1eb8 Add some new packages for future changes 2018-08-01 00:43:30 +01:00
Godzil
6f192b1712 Now use the config file for base configuration, and command line parameter to
overide some of these parameter.

The config file is updated from the command line parameters
2018-08-01 00:43:07 +01:00
Godzil
b947a110e2 Create config manager to store part of the config in a json file.
Update the IConfig structure to add new values for API login.
2018-08-01 00:40:20 +01:00
Godzil
68885db538 Add subtitle dump just in case. 2018-07-30 22:47:38 +01:00
Godzil
0b54549c64 Fix a silly bug in error management (in case the error does not come from request) 2018-07-30 22:47:38 +01:00
Godzil
141bdccf02 1.3.7 2018-07-30 22:47:38 +01:00
Godzil
4990effa1c Try to fix #81 and probably some other issues when the URL is not valid to properly display that the URL is not valid. Also change a bit on how error are handled 2018-07-30 22:47:38 +01:00
Godzil
2459f342c5 Force debug file to be written synchronously 2018-07-30 22:47:38 +01:00
Godzil
d68a2b7bce Update dependencies 2018-07-30 22:47:38 +01:00
Godzil
69d5ceac36 Remove useless ignore in .gitignore 2018-07-30 22:47:38 +01:00
Manoël Trapier
cf7039400c Update readme with more usefull examples 2018-07-30 22:47:38 +01:00
Godzil
02a9d763cd Add the episode title in the default file name template. 2018-07-30 22:47:38 +01:00
Godzil
d549d46979 1.3.6 2018-07-30 22:47:37 +01:00
Godzil
3f5b4b2585 Update readme 2018-07-30 22:47:37 +01:00
Godzil
1d596b02f7 Cleaning up the command line parameter to properly use default values 2018-07-30 22:47:37 +01:00
Godzil
cee53fb113 Fix for #78 (and a bit of cleanup) 2018-07-30 22:47:37 +01:00
Godzil
1e56cab73f Move error displaying when downloading an episode fail. 2018-07-30 22:47:37 +01:00
Godzil
0dc3c1e8e2 Update a bit the bug report template
(Commit #200!)
2018-07-30 22:47:37 +01:00
20 changed files with 797 additions and 1491 deletions

View File

@@ -14,10 +14,12 @@ A clear and concise description of what you expected to happen.
If applicable, add screenshots to help explain your problem.
**Please fill theses informations:**
(Add a X between brackets to make them ticked)
(Add a X between brackets to make them ticked if relevant)
- OS: [e.g:. Windows 10, Mac OS X 10.13, ...]
- [ ] I'm using the latest version of Crunchy
- [ ] I have a premium accrount on CR
- [ ] I am using a VPN
- My region in the world (country or continent):
- Serie you get a problem with (and specify which episode if it is specific to one):
- The command line you are running Crunchy with:
- The message Crunchy is giving you, if any:
@@ -29,4 +31,5 @@ If applicable, add screenshots to help explain your problem.
Add any other context about the problem here.
_Also don't hesitate to add labels you feel apropriate on your report._
_Also don't hesitate to add labels you feel apropriate on your report._
_Please don't edit logs if you are adding them, apart from removing sensitive informations like login/password_

2
.gitignore vendored
View File

@@ -1,3 +1,3 @@
dist/
node_modules/
typings/
package-lock.json

View File

@@ -3,6 +3,8 @@ sudo: false
node_js:
- 8
- 9
- 10
- 11
before_install:
- npm install --only=dev
script:

135
README.md
View File

@@ -1,6 +1,10 @@
# Crunchy: a fork of Deathspike/CrunchyRoll.js
[![Issue Stats](http://issuestats.com/github/Godzil/Crunchy/badge/issue)](http://issuestats.com/github/Godzil/Crunchy) [![Travis CI](https://travis-ci.org/Godzil/Crunchy.svg?branch=master)](https://travis-ci.org/Godzil/Crunchy) [![Maintainability](https://api.codeclimate.com/v1/badges/413c7ca11c0805b1ef3e/maintainability)](https://codeclimate.com/github/Godzil/Crunchy/maintainability)
[![Travis CI](https://travis-ci.org/Godzil/Crunchy.svg?branch=master)](https://travis-ci.org/Godzil/Crunchy)
[![Maintainability](https://api.codeclimate.com/v1/badges/413c7ca11c0805b1ef3e/maintainability)](https://codeclimate.com/github/Godzil/Crunchy/maintainability)
![npm](https://img.shields.io/npm/dy/crunchy.svg)
![Issue Count](https://img.shields.io/github/issues/Godzil/Crunchy.svg)
![npm](https://img.shields.io/npm/v/crunchy.svg?label=Last%20published%20version)
*Crunchy* is capable of downloading *anime* episodes from the popular *CrunchyRoll* streaming service. An episode is stored in the original video format (often H.264 in a MP4 container) and the configured subtitle format (ASS or SRT).The two output files are then merged into a single MKV file.
@@ -66,62 +70,120 @@ The [command-line interface](http://en.wikipedia.org/wiki/Command-line_interface
Options:
-V, --version output the version number
-p, --pass <s> The password.
-u, --user <s> The e-mail address or username.
-c, --cache Disables the cache.
-m, --merge Disables merging subtitles and videos.
-e, --episodes <s> Episode list. Read documentation on how to use
-f, --format <s> The subtitle format. (Default: ass)
-o, --output <s> The output path.
-s, --series <s> The series override.
-n, --filename <s> The name override.
-t, --tag <s> The subgroup. (Default: CrunchyRoll) (default: CrunchyRoll)
-r, --resolution <s> The video resolution. (Default: 1080 (360, 480, 720, 1080)) (default: 1080)
-g, --rebuildcrp Rebuild the crpersistant file.
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
--verbose Make tool verbose
--retry <i> Number or time to retry fetching an episode. Default: 5 (default: 5)
-h, --help output usage information
-V, --version output the version number
-p, --pass <s> The password.
-u, --user <s> The e-mail address or username.
-d, --unlog Unlog
-c, --cache Disables the cache.
-m, --merge Disables merging subtitles and videos.
-e, --episodes <s> Episode list. Read documentation on how to use
-f, --format <s> The subtitle format. (default: ass)
-o, --output <s> The output path.
-s, --series <s> The series name override.
--ignoredub Experimental: Ignore all seasons where the title end with 'Dub)'
-n, --nametmpl <s> Output name template (default: {SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - [{TAG}])
-t, --tag <s> The subgroup. (default: CrunchyRoll)
-r, --resolution <s> The video resolution. (valid: 360, 480, 720, 1080) (default: 1080)
-b, --batch <s> Batch file (default: CrunchyRoll.txt)
--verbose Make tool verbose
--rebuildcrp Rebuild the crpersistant file.
--retry <i> Number or time to retry fetching an episode. (default: 5)
-h, --help output usage information
#### Batch-mode
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed contain the URL of a series and can support some of the command line paramter (like *-e*). This makes it ideal to manage a large sequence of series addresses.
When no sequence of series addresses is provided, the batch-mode source file will be read (which is *CrunchyRoll.txt* in the current work directory. Each line in this file is processed contain the URL of a series and can support some of the command line parameter (like `-e`). This makes it ideal to manage a large sequence of series addresses.
#### Configuration file
Starting from version 1.4.0, Crunchy store some information in a config.json file. The file which is use have to be in the folder you are calling Crunchy. This is partly by design and a limitation on where Crunchy can find files.
This file store some informations like your username and password.
You don't need to create that file as Crunchy will create it for you, the first time you run it. Each run will update the content of the file, so it you run crunchy with your credential on the command line, it will add them to config file.
There are some parameter that the config file can accept which are not created by default, and some of them are cannont be set form the command line parameter.
Don't mess with them if you don't know what you are doing.
Here are the list of valid parameter in the config file:
- Output options
* `merge` see `--merge`
* `format` see `--format`
* `output` see `--output`
* `nametmpl` see `--nametmpl`
* `tag` see `--tag`
* `resolution` see `--resolution`
- Login related options:
* `pass` see `--user`
* `user` see `--pass`
* `userAgent` set the user agent reported by Crunchy while crawling pages
* `logUsingApi`
* `logUsingCookie`
* `crSessionUrl`
* `crDeviceType`
* `crAPIVersion`
* `crLocale`
* `crSessionKey`
* `crLoginUrl`
* `crUserId`
* `crUserKey`
- Generated values: don't touch them:
* `crDeviceId`
* `crSessionId`
Some of theses login related options are not going to be documented on what to put there for _legal_ reason.
Crunchy will also create a `.cookie.jar` file in the output folder (by default the current folder) it is the file used by Crunchy to store the web cookies.
#### Examples
Download in batch-mode:
crunchy
You will need to create the batch file (default name is `CrunchyRoll.txt`):
Download *Fairy Tail* to the current work directory:
http://www.cr.com/tail-fairy
http://www.cr.com/gin-mama
http://www.cr.com/two-parts
// Just download episodes 3 to 42
http://www.cr.com/defense-of-dwarfs -e 3-42
crunchy http://www.crunchyroll.com/fairy-tail
Then launch crunchy:
Download *Fairy Tail* to `C:\Anime`:
crunchy -u login -p password http://www.cr.com/tail-fairy
crunchy --output C:\Anime http://www.crunchyroll.com/fairy-tail
Download *Tail Fairy* to the current work directory:
Download episode 42 of *Fairy Tail* to `C:\Anime`:
crunchy -u login -p password http://www.cr.com/tail-fairy
crunchy --output C:\Anime @http://www.crunchyroll.com/fairy-tail/episode-46-the-silver-labyrinth-662721
Download *Tail Fairy* to `C:\Anime`:
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy
Download episode 42 of *Tail Fairy* to `C:\Anime`:
crunchy -u login -p password --output C:\Anime @http://www.cr.com/tail-fairy/episode-42-the-episode-which-dont-exist-665544
*Notice the '@' in front of the URL, it is there to tell Crunchy that the URL is an episode URL and not a series URL.*
or
crunchy --output C:\Anime http://www.crunchyroll.com/fairy-tail -e 42
Download episode 10 to 42 (both included) of *Fairy Tail*:
crunchy -u login -p password --output C:\Anime http://www.cr.com/tail-fairy -e 42
crunchy http://www.crunchyroll.com/fairy-tail -e 10-42
Download episode 10 to 42 (both included) of *Tail Fairy*:
Download episode up to 42 (included) of *Fairy Tail*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e 10-42
crunchy http://www.crunchyroll.com/fairy-tail -e -42
Download episode up to 42 (included) of *Tail Fairy*:
Download episodes starting from 42 to the last available of *Fairy Tail*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e -42
crunchy http://www.crunchyroll.com/fairy-tail -e 42-
Download episodes starting from 42 to the last available of *Tail Fairy*:
crunchy -u login -p password http://www.cr.com/tail-fairy -e 42-
@@ -131,8 +193,12 @@ Download episodes starting from 42 to the last available of *Fairy Tail*:
* `-p or --pass <s>` sets the password.
* `-u or --user <s>` sets the e-mail address or username.
* `-d or --unlog` unlog
_Please remember that login has to be done for each call of Crunchy, as none of the credentials are stored_
_New in 1.4.0_: Crunchy remember between run about login information and other, so you need to passe the login and password only once
I recommend to unlog if you see some problems during the run.
*When you unlog, the cookie file is deleted as for some parameter in the config file (like username and password).*
##### Disables
@@ -148,6 +214,7 @@ Download episodes starting from 42 to the last available of *Fairy Tail*:
* `-t or --tag <s>` sets The subgroup. (Default: CrunchyRoll)
* `-r or --resolution <s>` sets the resolutoin you want to download (360, 480, 720, 1080)
* `--retry <i>` set the number of try Crunchy will use if downloading a serie or episode fail
* `--ignoredub` It is an experimental features that will ignore all season where the name ends with 'dub)'. The idea is to try to ignore dubbed season.
##### Others

1186
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -15,36 +15,41 @@
"engines": {
"node": ">=5.0"
},
"version": "1.3.5",
"version": "1.5.0",
"bin": {
"crunchy": "./bin/crunchy",
"crunchy.sh": "./bin/crunchy.sh"
},
"dependencies": {
"@types/node": "^10.3.3",
"big-integer": "^1.6.31",
"bluebird": "^3.5.1",
"big-integer": "^1.6.44",
"bluebird": "^3.5.5",
"brotli": "^1.3.2",
"cheerio": "^0.22.0",
"cloudscraper": "^1.5.0",
"commander": "^2.15.1",
"fs-extra": "^6.0.1",
"cloudscraper": "^4.1.2",
"commander": "^2.20.0",
"fs-extra": "^8.1.0",
"mkdirp": "^0.5.0",
"pjson": "^1.0.9",
"request": "^2.87.0",
"request-promise": "^4.2.2",
"request": "^2.88.0",
"request-promise": "^4.2.4",
"tough-cookie-file-store": "^1.2.0",
"uuid": "^3.3.2",
"xml2js": "^0.4.5"
},
"devDependencies": {
"@types/bluebird": "^3.5.20",
"@types/cheerio": "^0.22.7",
"@types/fs-extra": "^5.0.3",
"@types/bluebird": "^3.5.27",
"@types/cheerio": "^0.22.12",
"@types/fs-extra": "^8.0.0",
"@types/mkdirp": "^0.5.2",
"@types/request": "^2.47.1",
"@types/request-promise": "^4.1.41",
"@types/xml2js": "^0.4.3",
"@types/node": "^12.6.8",
"@types/request": "^2.48.2",
"@types/request-promise": "^4.1.44",
"@types/uuid": "^3.4.5",
"@types/xml2js": "^0.4.4",
"npm-check": "^5.9.0",
"tsconfig-lint": "^0.12.0",
"tslint": "^5.10.0",
"typescript": "^2.9.2"
"tslint": "^5.18.0",
"typescript": "^3.5.3"
},
"scripts": {
"prepublishOnly": "npm run build",

View File

@@ -3,6 +3,8 @@ import commander = require('commander');
import fs = require('fs');
import path = require('path');
import log = require('./log');
import my_request = require('./my_request');
import cfg = require('./config');
import series from './series';
/* correspondances between resolution and value CR excpect */
@@ -19,8 +21,37 @@ const resol_table: { [id: string]: IResolData; } =
*/
export default function(args: string[], done: (err?: Error) => void)
{
const config = parse(args);
const batchPath = path.join(config.output || process.cwd(), config.batch);
const config = Object.assign(cfg.load(), parse(args));
let batchPath;
if (path.isAbsolute(config.batch))
{
batchPath = path.normalize(config.batch);
}
else
{
batchPath = path.normalize(path.join(process.cwd(), config.batch));
}
if (config.nametmpl === undefined)
{
config.nametmpl = '{SERIES_TITLE} - s{SEASON_NUMBER}e{EPISODE_NUMBER} - {EPISODE_TITLE} - [{TAG}]';
}
// Update the config file with new parameters
cfg.save(config);
if (config.unlog)
{
config.crDeviceId = undefined;
config.user = undefined;
config.pass = undefined;
my_request.eatCookies(config);
cfg.save(config);
log.info('Unlogged!');
process.exit(0);
}
// set resolution
if (config.resolution)
@@ -62,12 +93,20 @@ export default function(args: string[], done: (err?: Error) => void)
return done(err);
}
if (!tasksArr || !tasksArr[0] || (tasksArr[0].address === ''))
{
return done();
}
let i = 0;
(function next()
{
if (i >= tasksArr.length)
{
// Save configuration before leaving (should store info like session & other)
cfg.save(config);
return done();
}
@@ -80,12 +119,24 @@ export default function(args: string[], done: (err?: Error) => void)
{
if (errin)
{
if (tasksArr[i].retry <= 0)
if (errin.error)
{
log.error(errin.stack || errin);
/* Error from the request, so ignore it */
tasksArr[i].retry = 0;
}
if (errin.authError)
{
/* Force a graceful exit */
log.error(errin.message);
i = tasksArr.length;
}
else if (tasksArr[i].retry <= 0)
{
log.error(JSON.stringify(errin));
if (config.debug)
{
log.dumpToDebug('BatchGiveUp', errin.stack || errin);
log.dumpToDebug('BatchGiveUp', JSON.stringify(errin));
}
log.error('Cannot get episodes from "' + tasksArr[i].address + '", please rerun later');
/* Go to the next on the list */
@@ -95,11 +146,11 @@ export default function(args: string[], done: (err?: Error) => void)
{
if (config.verbose)
{
log.error(errin);
log.error(JSON.stringify(errin));
}
if (config.debug)
{
log.dumpToDebug('BatchRetry', errin.stack || errin);
log.dumpToDebug('BatchRetry', JSON.stringify(errin));
}
log.warn('Retrying to fetch episodes list from' + tasksArr[i].retry + ' / ' + config.retry);
tasksArr[i].retry -= 1;
@@ -195,6 +246,35 @@ function get_max_filter(filter: string): number
return +Infinity;
}
/**
* Check that URL start with http:// or https://
* As for some reason request just return an error but a useless one when that happen so check it
* soon enough.
*/
function checkURL(address: string): boolean
{
if (address.startsWith('http:\/\/'))
{
return true;
}
if (address.startsWith('https:\/\/'))
{
return true;
}
if (address.startsWith('@http:\/\/'))
{
return true;
}
if (address.startsWith('@https:\/\/'))
{
return true;
}
log.error('URL ' + address + ' miss \'http:\/\/\' or \'https:\/\/\' => will be ignored');
return false;
}
/**
* Parses the configuration or reads the batch-mode file for tasks.
*/
@@ -204,8 +284,13 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
{
return done(null, config.args.map((addressIn) =>
{
return {address: addressIn, retry: config.retry,
episode_min: get_min_filter(config.episodes), episode_max: get_max_filter(config.episodes)};
if (checkURL(addressIn))
{
return {address: addressIn, retry: config.retry,
episode_min: get_min_filter(config.episodes), episode_max: get_max_filter(config.episodes)};
}
return {address: '', retry: 0, episode_min: 0, episode_max: 0};
}));
}
@@ -241,8 +326,11 @@ function tasks(config: IConfigLine, batchPath: string, done: (err: Error, tasks?
return;
}
map.push({address: addressIn, retry: lineConfig.retry,
episode_min: get_min_filter(lineConfig.episodes), episode_max: get_max_filter(lineConfig.episodes)});
if (checkURL(addressIn))
{
map.push({address: addressIn, retry: lineConfig.retry,
episode_min: get_min_filter(lineConfig.episodes), episode_max: get_max_filter(lineConfig.episodes)});
}
});
});
done(null, map);
@@ -259,23 +347,25 @@ function parse(args: string[]): IConfigLine
// Authentication
.option('-p, --pass <s>', 'The password.')
.option('-u, --user <s>', 'The e-mail address or username.')
.option('-d, --unlog', 'Unlog')
// Disables
.option('-c, --cache', 'Disables the cache.')
.option('-m, --merge', 'Disables merging subtitles and videos.')
// Episode filter
.option('-e, --episodes <s>', 'Episode list. Read documentation on how to use')
// Settings
.option('-f, --format <s>', 'The subtitle format. (Default: ass)')
.option('-l, --crlang <s>', 'CR page language (valid: en, fr, es, it, pt, de, ru).')
.option('-f, --format <s>', 'The subtitle format.', 'ass')
.option('-o, --output <s>', 'The output path.')
.option('-s, --series <s>', 'The series override.')
.option('-n, --filename <s>', 'The name override.')
.option('-t, --tag <s>', 'The subgroup. (Default: CrunchyRoll)', 'CrunchyRoll')
.option('-r, --resolution <s>', 'The video resolution. (Default: 1080 (360, 480, 720, 1080))',
'1080')
.option('-g, --rebuildcrp', 'Rebuild the crpersistant file.')
.option('-s, --series <s>', 'The series name override.')
.option('--ignoredub', 'Experimental: Ignore all seasons where the title end with \'Dub)\'')
.option('-n, --nametmpl <s>', 'Output name template')
.option('-t, --tag <s>', 'The subgroup.', 'CrunchyRoll')
.option('-r, --resolution <s>', 'The video resolution. (valid: 360, 480, 720, 1080)', '1080')
.option('-b, --batch <s>', 'Batch file', 'CrunchyRoll.txt')
.option('--verbose', 'Make tool verbose')
.option('--debug', 'Create a debug file. Use only if requested!')
.option('--retry <i>', 'Number or time to retry fetching an episode. Default: 5', 5)
.option('--rebuildcrp', 'Rebuild the crpersistant file.')
.option('--retry <i>', 'Number or time to retry fetching an episode.', 5)
.parse(args);
}

View File

@@ -8,17 +8,27 @@ const current_version = pjson.version;
/* Check if the current version is the latest */
log.info('Crunchy version ' + current_version);
request.get({ uri: 'https://raw.githubusercontent.com/Godzil/Crunchy/master/package.json' },
request.get({ uri: 'https://box.godzil.net/getVersion.php?tool=crunchy&v=' + current_version },
(error: Error, response: any, body: any) =>
{
const onlinepkg = JSON.parse(body);
let tmp = current_version.split('.');
const cur = (Number(tmp[0]) * 10000) + (Number(tmp[1]) * 100) + Number(tmp[2]);
tmp = onlinepkg.version.split('.');
const dist = (Number(tmp[0]) * 10000) + (Number(tmp[1]) * 100) + Number(tmp[2]);
if (dist > cur)
if (response && (response.statusCode === 200))
{
log.warn('There is a newer version of crunchy (v' + onlinepkg.version + '), you should update!');
const onlinepkg = JSON.parse(body);
if (onlinepkg.status === 'ok')
{
let tmp = current_version.split('.');
const cur = (Number(tmp[0]) * 10000) + (Number(tmp[1]) * 100) + Number(tmp[2]);
tmp = onlinepkg.version.split('.');
const dist = (Number(tmp[0]) * 10000) + (Number(tmp[1]) * 100) + Number(tmp[2]);
if (dist > cur)
{
log.warnMore('There is a newer version of crunchy (v' + onlinepkg.version + '), you should update!');
}
}
}
else
{
log.info('Error while checking for the current version.');
}
});

63
src/config.ts Normal file
View File

@@ -0,0 +1,63 @@
'use strict';
import os = require('os');
import fs = require('fs-extra');
import path = require('path');
const configFile = path.join(process.cwd(), 'config.json');
function fileExist(path: string)
{
try
{
fs.statSync(path);
return true;
} catch (e)
{
return false;
}
}
export function load(): IConfigLine
{
if (fileExist(configFile))
{
const data = fs.readFileSync(configFile, 'utf8');
return JSON.parse(data);
}
return {args: undefined};
}
export function save(config: IConfig)
{
const tmp = JSON.parse(JSON.stringify(config));
// Things added by the command line parser
tmp.rawArgs = undefined;
tmp.options = undefined;
tmp._execs = undefined;
tmp._args = undefined;
tmp._name = undefined;
tmp._version = undefined;
tmp._versionOptionName = undefined;
tmp._events = undefined;
tmp._eventsCount = undefined;
tmp.args = undefined;
tmp.commands = undefined;
tmp._allowUnknownOption = undefined;
// Things we don't want to save
tmp.cache = undefined;
tmp.episodes = undefined;
tmp.series = undefined;
tmp.video_format = undefined;
tmp.video_quality = undefined;
tmp.rebuildcrp = undefined;
tmp.batch = undefined;
tmp.verbose = undefined;
tmp.debug = undefined;
tmp.unlog = undefined;
tmp.ignoredub = undefined;
fs.writeFileSync(configFile, JSON.stringify(tmp, null, ' '));
}

View File

@@ -66,7 +66,7 @@ function fileExist(path: string)
function sanitiseFileName(str: string)
{
return str.replace(/[\/':\?\*"<>\.]/g, '_');
return str.replace(/[\/':\?\*"<>\\\.\|]/g, '_');
}
/**
@@ -74,11 +74,10 @@ function sanitiseFileName(str: string)
*/
function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, done: (err: Error, ign: boolean) => void)
{
let series = config.series || page.series;
const serieFolder = sanitiseFileName(config.series || page.series);
series = sanitiseFileName(series);
let fileName = sanitiseFileName(name(config, page, series, ''));
let filePath = path.join(config.output || process.cwd(), series, fileName);
let fileName = sanitiseFileName(generateName(config, page));
let filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
if (fileExist(filePath + '.mkv'))
{
@@ -95,8 +94,8 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
do
{
count = count + 1;
fileName = sanitiseFileName(name(config, page, series, '-' + count));
filePath = path.join(config.output || process.cwd(), series, fileName);
fileName = sanitiseFileName(generateName(config, page, '-' + count));
filePath = path.join(config.output || process.cwd(), serieFolder, fileName);
} while (fileExist(filePath + '.mkv'));
log.warn('Renaming to \'' + fileName + '\'...');
@@ -114,6 +113,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errM)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errM, false);
}
@@ -122,6 +122,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errDS)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errDS, false);
}
@@ -133,6 +134,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errDV)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errDV, false);
}
@@ -148,6 +150,7 @@ function download(config: IConfig, page: IEpisodePage, player: IEpisodePlayer, d
{
if (errVM)
{
log.dispEpisode(fileName, 'Error...', true);
return done(errVM, false);
}
@@ -183,6 +186,11 @@ function downloadSubtitle(config: IConfig, player: IEpisodePlayer, filePath: str
return done(errSD);
}
if (config.debug)
{
log.dumpToDebug('SubtitlesXML', data);
}
const formats = subtitle.formats;
const format = formats[config.format] ? config.format : 'ass';
@@ -211,19 +219,16 @@ function downloadVideo(config: IConfig, page: IEpisodePage, player: IEpisodePla
/**
* Names the file based on the config, page, series and tag.
*/
function name(config: IConfig, page: IEpisodePage, series: string, extra: string)
function generateName(config: IConfig, page: IEpisodePage, extra = '')
{
const episodeNum = parseInt(page.episode, 10);
const volumeNum = parseInt(page.volume, 10);
const episode = (episodeNum < 10 ? '0' : '') + page.episode;
const volume = (volumeNum < 10 ? '0' : '') + page.volume;
const tag = config.tag || 'CrunchyRoll';
const series = config.series || page.series;
if (!page.filename) {
return page.series + ' - s' + volume + 'e' + episode + ' - [' + tag + ']' + extra;
}
return page.filename
return config.nametmpl
.replace(/{EPISODE_ID}/g, page.id.toString())
.replace(/{EPISODE_NUMBER}/g, episode)
.replace(/{SEASON_NUMBER}/g, volume)

3
src/interface/AuthError.d.ts vendored Normal file
View File

@@ -0,0 +1,3 @@
interface IAuthError extends Error {
authError: boolean;
}

View File

@@ -7,11 +7,13 @@ interface IConfig {
merge?: boolean;
episodes?: string;
// Settings
crlang?: string;
format?: string;
output?: string;
series?: string;
filename?: string;
nametmpl?: string;
tag?: string;
ignoredub?: boolean;
resolution?: string;
video_format?: string;
video_quality?: string;
@@ -19,5 +21,22 @@ interface IConfig {
batch?: string;
verbose?: boolean;
debug?: boolean;
unlog?: boolean;
retry?: number;
// Login options
userAgent?: string;
logUsingApi?: boolean;
logUsingCookie?: boolean;
crSessionUrl?: string;
crDeviceType?: string;
crAPIVersion?: string;
crLocale?: string;
crSessionKey?: string;
crLoginUrl?: string;
// Third method, injecting data from cookies
crUserId?: string;
crUserKey?: string;
// Generated values
crDeviceId?: string;
crSessionId?: string;
}

View File

@@ -1,6 +1,7 @@
interface ISeriesEpisode {
address: string;
episode: string;
seasonName: string;
volume: number;
retry: number;
}

66
src/languages.ts Normal file
View File

@@ -0,0 +1,66 @@
'use strict';
const localeCC: { [id: string]: string; } =
{
enUS: 'en', enGB: 'en',
esLA: 'es', esES: 'es',
ptPT: 'pt', ptBR: 'pt',
frFR: 'fr',
deDE: 'de',
itIT: 'it',
ruRU: 'ru',
};
export function localeToCC(locale: string): string
{
let ret = localeCC.enGB;
if (locale in localeCC)
{
ret = localeCC[locale];
}
return ret;
}
const dubignore_regexp: { [id: string]: RegExp; } =
{
en: /\(.*Dub(?:bed)?.*\)|(?:\(RU\))/i,
fr: /\(.*Dub(?:bed)?.*\)|(?:\(RU\))|\(?Doublage.*\)?/,
de: /\(.*isch\)|\(Dubbed\)|\(RU\)/
};
export function get_diregexp(config: IConfig): RegExp
{
let ret = dubignore_regexp.en;
if (config.crlang in dubignore_regexp)
{
ret = dubignore_regexp[config.crlang];
}
return ret;
}
const episodes_regexp: { [id: string]: RegExp; } =
{
en: /Episode\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
fr: /Épisode\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
de: /Folge\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
es: /Episodio\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
it: /Episodio\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
pt: /Episódio\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
ru: /Серия\s+((OVA)|(PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i,
};
export function get_epregexp(config: IConfig): RegExp
{
let ret = episodes_regexp.en;
if (config.crlang in episodes_regexp)
{
ret = episodes_regexp[config.crlang];
}
return ret;
}

View File

@@ -26,6 +26,12 @@ export function warn(str: string)
console.log(' \x1B[1;33m* WARN \x1B[0m: ' + str);
}
export function warnMore(str: string)
{
/* Do fancy output */
console.log(' \x1B[1;38;5;166m* WARN \x1B[0m: ' + str);
}
export function dispEpisode(name: string, status: string, addNL: boolean)
{
/* Do fancy output */
@@ -41,15 +47,9 @@ export function dumpToDebug(what: string, data: any, create = false)
{
if (create)
{
fs.writeFile('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n', (err) =>
{
if (err) throw err;
});
fs.writeFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
return;
}
fs.appendFile('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n', (err) =>
{
if (err) throw err;
});
}
fs.appendFileSync('debug.txt', '>>>>>>>> ' + what + ':\n' + data + '\n<<<<<<<<\n');
}

View File

@@ -3,72 +3,93 @@ import cheerio = require('cheerio');
import request = require('request');
import rp = require('request-promise');
import Promise = require('bluebird');
import uuid = require('uuid');
import path = require('path');
import fs = require('fs-extra');
import languages = require('./languages');
import log = require('./log');
import { RequestPromise } from 'request-promise';
import { Response } from 'request';
// tslint:disable-next-line:no-var-requires
const cloudscraper = require('cloudscraper');
const cookieStore = require('tough-cookie-file-store');
const CR_COOKIE_DOMAIN = 'http://crunchyroll.com';
let isAuthenticated = false;
let isPremium = false;
const defaultHeaders: request.Headers =
let j: request.CookieJar;
const defaultHeaders =
{
'User-Agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64; x64; rv:58.0) Gecko/20100101 Firefox/58.0',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',
'Connection': 'keep-alive',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
'Referer': 'https://www.crunchyroll.com/login',
'Cache-Control': 'private',
'Accept-Language': 'en-US,en;q=0.9'
};
function generateDeviceId(): string
const defaultOptions =
{
let id = '';
const possible = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
followAllRedirects: true,
decodeEmails: true,
challengesToSolve: 3,
gzip: true,
};
for (let i = 0; i < 32; i++)
{
id += possible.charAt(Math.floor(Math.random() * possible.length));
}
// tslint:disable-next-line:no-var-requires
const cloudscraper = require('cloudscraper').defaults(defaultOptions);
return id;
function AuthError(msg: string): IAuthError
{
return { name: 'AuthError', message: msg, authError: true };
}
function startSession(): Promise<string>
function startSession(config: IConfig): Promise<any>
{
return rp(
{
method: 'GET',
url: 'CR_SESSION_URL',
url: config.crSessionUrl,
qs:
{
device_id: generateDeviceId(),
device_type: 'CR_DEVICE_TYPE',
access_token: 'CR_SESSION_KEY',
version: 'CR_API_VERSION',
locale: 'CR_LOCALE',
device_id: config.crDeviceId,
device_type: config.crDeviceType,
access_token: config.crSessionKey,
version: config.crAPIVersion,
locale: config.crLocale,
},
json: true,
})
.then((response: any) =>
{
if ((response.data === undefined) || (response.data.session_id === undefined))
{
throw new Error('Getting session failed: ' + JSON.stringify(response));
}
return response.data.session_id;
});
}
function login(sessionId: string, user: string, pass: string): Promise<any>
function APIlogin(config: IConfig, sessionId: string, user: string, pass: string): Promise<any>
{
return rp(
{
method: 'POST',
url: 'CR_LOGIN_URL',
url: config.crLoginUrl,
form:
{
account: user,
password: pass,
session_id: sessionId,
version: 'CR_API_VERSION',
version: config.crAPIVersion,
},
json: true,
jar: j,
})
.then((response) =>
{
@@ -77,13 +98,130 @@ function login(sessionId: string, user: string, pass: string): Promise<any>
});
}
// TODO: logout
function checkIfUserIsAuth(config: IConfig, done: (err: Error) => void): void
{
if (j === undefined)
{
loadCookies(config);
}
/**
* The main page give us some information about the user
*/
const url = 'http://www.crunchyroll.com/';
cloudscraper.get({gzip: true, uri: url}, (err: Error, rep: string, body: string) =>
{
if (err)
{
return done(err);
}
const $ = cheerio.load(body);
/* As we are here, try to detect which locale CR tell us */
const localeRE = /LOCALE = "([a-zA-Z]+)",/g;
const locale = localeRE.exec($('script').text())[1];
const countryCode = languages.localeToCC(locale);
if (config.crlang === undefined)
{
log.info('No locale set. Setting to the one reported by CR: "' + countryCode + '"');
config.crlang = countryCode;
}
else if (config.crlang !== countryCode)
{
log.warn('Crunchy is configured for locale "' + config.crlang + '" but CR report "' + countryCode + '" (LOCALE = ' + locale + ')');
log.warn('Check if it is correct or rerun (once) with "-l ' + countryCode + '" to correct.');
}
/* Check if auth worked */
const regexps = /ga\('set', 'dimension[5-8]', '([^']*)'\);/g;
const dims = regexps.exec($('script').text());
for (let i = 1; i < 5; i++)
{
if ((dims[i] !== undefined) && (dims[i] !== '') && (dims[i] !== 'not-registered'))
{
isAuthenticated = true;
}
if ((dims[i] === 'premium') || (dims[i] === 'premiumplus'))
{
isPremium = true;
}
}
if (isAuthenticated === false)
{
const error = $('ul.message, li.error').text();
log.warn('Authentication failed: ' + error);
log.dumpToDebug('not auth rep', rep);
log.dumpToDebug('not auth body', body);
return done(AuthError('Authentication failed: ' + error));
}
else
{
if (isPremium === false)
{
log.warn('Do not use this app without a premium account.');
}
else
{
log.info('You have a premium account! Good!');
}
}
done(null);
});
}
function loadCookies(config: IConfig)
{
const cookiePath = path.join(config.output || process.cwd(), '.cookies.json');
if (!fs.existsSync(cookiePath))
{
fs.closeSync(fs.openSync(cookiePath, 'w'));
}
j = request.jar(new cookieStore(cookiePath));
}
export function eatCookies(config: IConfig)
{
const cookiePath = path.join(config.output || process.cwd(), '.cookies.json');
if (fs.existsSync(cookiePath))
{
fs.removeSync(cookiePath);
}
j = undefined;
}
export function getUserAgent(): string
{
return defaultHeaders['User-Agent'];
}
/**
* Performs a GET request for the resource.
*/
export function get(config: IConfig, options: string|request.Options, done: (err: Error, result?: string) => void)
export function get(config: IConfig, options: string|any, done: (err: any, result?: string) => void)
{
if (j === undefined)
{
loadCookies(config);
}
if (config.userAgent)
{
defaultHeaders['User-Agent'] = config.userAgent;
}
authenticate(config, (err) =>
{
if (err)
@@ -91,9 +229,10 @@ export function get(config: IConfig, options: string|request.Options, done: (err
return done(err);
}
cloudscraper.request(modify(options, 'GET'), (error: Error, response: any, body: any) =>
cloudscraper.get(modify(options), (error: any, response: any, body: any) =>
{
if (error) return done(error);
done(null, typeof body === 'string' ? body : String(body));
});
});
@@ -104,18 +243,28 @@ export function get(config: IConfig, options: string|request.Options, done: (err
*/
export function post(config: IConfig, options: request.Options, done: (err: Error, result?: string) => void)
{
if (j === undefined)
{
loadCookies(config);
}
if (config.userAgent)
{
defaultHeaders['User-Agent'] = config.userAgent;
}
authenticate(config, (err) =>
{
if (err)
{
return done(err);
return done(err);
}
cloudscraper.request(modify(options, 'POST'), (error: Error, response: any, body: any) =>
cloudscraper.post(modify(options), (error: Error, response: any, body: any) =>
{
if (error)
{
return done(error);
return done(error);
}
done(null, typeof body === 'string' ? body : String(body));
});
@@ -132,95 +281,153 @@ function authenticate(config: IConfig, done: (err: Error) => void)
return done(null);
}
if (!config.pass || !config.user)
/* First of all, check if the user is not already logged via the cookies */
checkIfUserIsAuth(config, (errCheckAuth) =>
{
log.error('You need to give login/password to use Crunchy');
process.exit(-1);
}
startSession()
.then((sessionId: string) =>
{
defaultHeaders.Cookie = `sess_id=${sessionId}; c_locale=enUS`;
return login(sessionId, config.user, config.pass);
})
.then((userData) =>
{
/**
* The page return with a meta based redirection, as we wan't to check that everything is fine, reload
* the main page. A bit convoluted, but more sure.
*/
const options =
if (isAuthenticated)
{
headers: defaultHeaders,
jar: true,
url: 'http://www.crunchyroll.com/',
method: 'GET',
};
return done(null);
}
cloudscraper.request(options, (err: Error, rep: string, body: string) =>
/* So if we are here now, that mean we are not authenticated so do as usual */
if ((!config.logUsingApi && !config.logUsingCookie) && (!config.pass || !config.user))
{
if (err)
log.error('You need to give login/password to use Crunchy');
process.exit(-1);
}
log.info('Seems we are not currently logged. Let\'s login!');
if (config.logUsingApi)
{
if (config.crDeviceId === undefined)
{
return done(err);
config.crDeviceId = uuid.v4();
}
const $ = cheerio.load(body);
/* Check if auth worked */
const regexps = /ga\('set', 'dimension[5-8]', '([^']*)'\);/g;
const dims = regexps.exec($('script').text());
for (let i = 1; i < 5; i++)
if (!config.crSessionUrl || !config.crDeviceType || !config.crAPIVersion ||
!config.crLocale || !config.crLoginUrl)
{
if ((dims[i] !== undefined) && (dims[i] !== '') && (dims[i] !== 'not-registered'))
return done(AuthError('Invalid API configuration, please check your config file.'));
}
startSession(config)
.then((sessionId: string) =>
{
// defaultHeaders['Cookie'] = `sess_id=${sessionId}; c_locale=enUS`;
return APIlogin(config, sessionId, config.user, config.pass);
})
.then((userData) =>
{
checkIfUserIsAuth(config, (errCheckAuth2) =>
{
isAuthenticated = true;
if (isAuthenticated)
{
return done(null);
}
else
{
return done(errCheckAuth2);
}
});
})
.catch((errInChk) =>
{
return done(AuthError(errInChk.message));
});
}
else if (config.logUsingCookie)
{
j.setCookie(request.cookie('c_userid=' + config.crUserId + '; Domain=crunchyroll.com; HttpOnly; hostOnly=false;'),
CR_COOKIE_DOMAIN);
j.setCookie(request.cookie('c_userkey=' + config.crUserKey + '; Domain=crunchyroll.com; HttpOnly; hostOnly=false;'),
CR_COOKIE_DOMAIN);
checkIfUserIsAuth(config, (errCheckAuth2) =>
{
if (isAuthenticated)
{
return done(null);
}
else
{
return done(errCheckAuth2);
}
});
}
else
{
/* First get https://www.crunchyroll.com/login to get the login token */
const options =
{
// jar: j,
uri: 'https://www.crunchyroll.com/login'
};
cloudscraper.get(options, (err: Error, rep: string, body: string) =>
{
if (err) return done(err);
const $ = cheerio.load(body);
/* Get the token from the login page */
const token = $('input[name="login_form[_token]"]').attr('value');
if (token === '')
{
return done(AuthError('Can\'t find token!'));
}
if ((dims[i] === 'premium') || (dims[i] === 'premiumplus'))
/* Now call the page again with the token and credentials */
const options =
{
isPremium = true;
}
}
form:
{
'login_form[name]': config.user,
'login_form[password]': config.pass,
'login_form[redirect_url]': '/',
'login_form[_token]': token
},
// jar: j,
url: 'https://www.crunchyroll.com/login'
};
if (isAuthenticated === false)
{
const error = $('ul.message, li.error').text();
log.error('Authentication failed: ' + error);
process.exit(-1);
}
cloudscraper.post(options, (err: Error, rep: string, body: string) =>
{
if (err)
{
return done(err);
}
if (isPremium === false)
{
log.warn('Do not use this app without a premium account.');
}
else
{
log.info('You have a premium account! Good!');
}
done(null);
});
})
.catch(done);
/* Now let's check if we are authentificated */
checkIfUserIsAuth(config, (errCheckAuth2) =>
{
if (isAuthenticated)
{
return done(null);
}
else
{
return done(errCheckAuth2);
}
});
});
});
}
});
}
/**
* Modifies the options to use the authenticated cookie jar.
*/
function modify(options: string|request.Options, reqMethod: string): request.Options
function modify(options: string|any): any
{
if (typeof options !== 'string')
{
options.jar = true;
options.headers = defaultHeaders;
options.method = reqMethod;
options.jar = j;
return options;
}
return {
jar: true,
headers: defaultHeaders,
jar: j,
url: options.toString(),
method: reqMethod
};
}
}

View File

@@ -1,12 +1,13 @@
'use strict';
import cheerio = require('cheerio');
import episode from './episode';
// import fs = require('fs');
import fs = require('fs-extra');
import my_request = require('./my_request');
import path = require('path');
import url = require('url');
import log = require('./log');
import languages = require('./languages');
const persistent = '.crpersistent';
/**
@@ -27,7 +28,7 @@ function fileExist(path: string)
/**
* Streams the series to disk.
*/
export default function(config: IConfig, task: IConfigTask, done: (err: Error) => void)
export default function(config: IConfig, task: IConfigTask, done: (err: any) => void)
{
const persistentPath = path.join(config.output || process.cwd(), persistent);
@@ -45,6 +46,15 @@ export default function(config: IConfig, task: IConfigTask, done: (err: Error) =
{
if (errP)
{
const reqErr = errP.error;
if ((reqErr !== undefined) && (reqErr.syscall))
{
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
{
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
}
}
return done(errP);
}
@@ -61,10 +71,20 @@ export default function(config: IConfig, task: IConfigTask, done: (err: Error) =
{
if (errD)
{
/* Check if domain is valid */
const reqErr = errD.error;
if ((reqErr !== undefined) && (reqErr.syscall))
{
if ((reqErr.syscall === 'getaddrinfo') && (reqErr.errno === 'ENOTFOUND'))
{
page.episodes[i].retry = 0;
log.error('The URL \'' + task.address + '\' is invalid, please check => I\'m ignoring it.');
}
}
if (page.episodes[i].retry <= 0)
{
log.dispEpisode(config.filename, 'Error...', true);
log.error(errD);
log.error(JSON.stringify(errD));
log.error('Cannot fetch episode "s' + page.episodes[i].volume + 'e' + page.episodes[i].episode +
'", please rerun later');
/* Go to the next on the list */
@@ -72,13 +92,12 @@ export default function(config: IConfig, task: IConfigTask, done: (err: Error) =
}
else
{
log.dispEpisode(config.filename, 'Error...', true);
if ((config.verbose) || (config.debug))
{
if (config.debug)
{
log.dumpToDebug('series address', task.address);
log.dumpToDebug('series error', errD.stack || errD);
log.dumpToDebug('series error', JSON.stringify(errD));
log.dumpToDebug('series data', JSON.stringify(page));
}
log.error(errD);
@@ -122,7 +141,7 @@ export default function(config: IConfig, task: IConfigTask, done: (err: Error) =
*/
function download(cache: {[address: string]: number}, config: IConfig,
task: IConfigTask, item: ISeriesEpisode,
done: (err: Error, ign: boolean) => void)
done: (err: any, ign: boolean) => void)
{
const episodeNumber = parseInt(item.episode, 10);
@@ -154,7 +173,7 @@ function download(cache: {[address: string]: number}, config: IConfig,
/**
* Requests the page and scrapes the episodes and series.
*/
function pageScrape(config: IConfig, task: IConfigTask, done: (err: Error, result?: ISeries) => void)
function pageScrape(config: IConfig, task: IConfigTask, done: (err: any, result?: ISeries) => void)
{
if (task.address[0] === '@')
{
@@ -163,6 +182,7 @@ function pageScrape(config: IConfig, task: IConfigTask, done: (err: Error, resul
episodes.push({
address: task.address.substr(1),
episode: '',
seasonName: '',
volume: 0,
retry: config.retry,
});
@@ -172,12 +192,13 @@ function pageScrape(config: IConfig, task: IConfigTask, done: (err: Error, resul
{
let episodeCount = 0;
my_request.get(config, task.address, (err, result) => {
if (err) {
if (err)
{
return done(err);
}
const $ = cheerio.load(result);
const title = $('span[itemprop=name]').text();
const title = $('meta[itemprop=name]').attr('content');
if (config.debug)
{
@@ -195,27 +216,50 @@ function pageScrape(config: IConfig, task: IConfigTask, done: (err: Error, resul
log.info('Checking availability for ' + title);
const episodes: ISeriesEpisode[] = [];
if ($('.availability-notes-low').length)
{
log.warn('This serie may have georestriction and some missings episode (like some dubs)' +
' [Message: ' + $('.availability-notes-low').text() + '].');
}
if ($('.availability-notes-high').length)
{
log.warnMore('This serie probably have georestriction and will miss some episodes' +
' [Message: ' + $('.availability-notes-high').text() + '].');
}
$('.episode').each((i, el) => {
if ($(el).children('img[src*=coming_soon]').length) {
return;
}
const season_name = $(el).closest('ul').prev('a').text();
const volume = /([0-9]+)\s*$/.exec($(el).closest('ul').prev('a').text());
const regexp = /Episode\s+((PV )?[S0-9][\-P0-9.]*[a-fA-F]?)\s*$/i;
const regexp = languages.get_epregexp(config);
const episode = regexp.exec($(el).children('.series-title').text());
const url = $(el).attr('href');
if ((!url) || (!episode)) {
const igndub_re = languages.get_diregexp(config);
if (config.ignoredub && (igndub_re.exec(season_name)))
{
return;
}
if ((!url) || (!episode))
{
return;
}
episodeCount += 1;
episodes.push({
address: url,
episode: episode[1],
seasonName: season_name,
volume: volume ? parseInt(volume[0], 10) : 1,
retry: config.retry,
});
});
if (episodeCount === 0)
{
log.warn('No episodes found for ' + title + '. Could it be a movie?');

View File

@@ -3,6 +3,7 @@ import childProcess = require('child_process');
import os = require('os');
import path = require('path');
import my_request = require('../my_request');
import log = require('../log');
/**
@@ -24,7 +25,8 @@ export default function(rtmpUrl: string, rtmpInputPath: string, swfUrl: string,
else if (mode === 'HLS')
{
cmd = command('ffmpeg') + ' ' +
'-y -xerror ' +
'-user_agent "' + my_request.getUserAgent() + '" ' +
'-y -xerror -discard none ' +
'-i "' + rtmpInputPath + '" ' +
'-c copy -bsf:a aac_adtstoasc ' +
'"' + filePath + '.mp4"';

96
ts.js
View File

@@ -1,96 +0,0 @@
'use strict';
var childProcess = require('child_process');
var fs = require('fs');
var path = require('path');
var isTest = process.argv[2] === '--only-test';
// TODO: This build task should be removed upon release of TypeScript 1.5 with
// the support for `tsconfig.json`. Invoking `tsc` from `package.json` will then
// read the configuration and compile accordingly. It seems that `TSLint` will,
// eventually, support this mechanism too. That prevents the need for any kind
// of build task and will run entirely based on instructions from `npm`.
//
// Reference #1: https://github.com/Microsoft/TypeScript/issues/1667
// Reference #2: https://github.com/palantir/tslint/issues/281
read(function(err, fileNames) {
clean(fileNames, function() {
var hasLintError = false;
compile(fileNames, function(err) {
if (err) {
console.error(err);
return process.exit(1);
}
lint(fileNames, function(message) {
process.stdout.write(message);
hasLintError = true;
}, function() {
process.exit(Number(hasLintError));
});
});
});
});
/**
* Clean the files.
* @param {Array.<string>} filePaths
* @param {function()} done
*/
function clean(filePaths, done) {
if (isTest) return done();
var i = -1;
(function next() {
i += 1;
if (i >= filePaths.length) return done();
var filePath = filePaths[i];
if (/\.d\.ts$/.test(filePath)) return next();
var mapName = filePath.substring(4, filePath.length - 2) + 'js.map';
var mapPath = path.join('dist', mapName);
if (fs.existsSync(mapPath)) fs.unlinkSync(mapPath);
next();
})();
}
/**
* Compile the files.
* @param {Array.<string>} filePaths
* @param {function(Error)} done
*/
function compile(filePaths, done) {
if (isTest) return done(null);
var execPath = path.join(__dirname, 'node_modules/.bin/tsc');
var options = '--declaration --module CommonJS --noImplicitAny --outDir dist --target ES5';
childProcess.exec([execPath, options].concat(filePaths).join(' '), function(err, stdout) {
if (stdout) return done(new Error(stdout));
done(null);
});
}
/**
* Lint the files.
* @param {Array.<string>} filePaths
* @param {function(string)} handler
* @param {function()} done
*/
function lint(filePaths, handler, done) {
var i = -1;
var execPath = path.join(__dirname, 'node_modules/.bin/tslint');
(function next() {
i += 1;
if (i >= filePaths.length) return done();
var filePath = filePaths[i];
if (/\.d\.ts$/.test(filePath)) return next();
childProcess.exec(execPath + ' -f ' + filePath, function(err, stdout) {
if (stdout) handler(stdout);
next();
});
})();
}
/**
* Read the files from the project file.
* @param {function(Error, Array.<string>)} done
*/
function read(done) {
done(null, JSON.parse(fs.readFileSync('tsconfig.json', 'utf8')).files);
}

View File

@@ -21,6 +21,7 @@
"no-any": false,
"no-arg": true,
"no-bitwise": true,
"prefer-conditional-expression": false,
"space-within-parens": false,
"no-object-literal-type-assertion": false,
"no-console": [true,