Skip to content
GitLab
Explore
Projects
Groups
Snippets
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
yujiosaka
headless-chrome-crawler
Merge requests
!45
An error occurred while fetching the assigned milestone of the selected merge_request.
Fix a bug of opening too many tabs when error occured
Code
Review changes
Check out branch
Download
Email patches
Plain diff
Merged
yujiosaka
requested to merge
fix_too_many_tabs
into
master
7 years ago
Overview
1
Commits
1
Pipelines
0
Changes
1
Fix a bug of opening too many tabs at the same time
0
0
Compare
master
master (base)
and
latest version
latest version
1e312172
1 commit,
2 years ago
1 file
+
18
-
16
Expand all files
Preferences
Preferences
File browser
List view
Tree view
Compare changes
Inline
Side-by-side
Show whitespace changes
Show one file at a time
lib/hccrawler.js
+
18
-
16
Options
View file @ 1e312172
Edit in single-file editor
Open in Web IDE
Show full file
@@ -265,25 +265,27 @@ class HCCrawler {
links
:
res
.
links
,
options
,
}))
.
then
(()
=>
void
debugRequest
(
`End requesting
${
this
.
_options
.
url
}
`
))
.
then
(()
=>
crawler
.
close
())
.
then
(()
=>
void
this
.
_checkRequestCount
())
.
then
(()
=>
delay
(
options
.
delay
))
.
then
(()
=>
this
.
_followLinks
(
res
.
links
,
options
,
depth
))
))
))
.
then
(()
=>
void
debugRequest
(
`End requesting
${
this
.
_options
.
url
}
`
))
.
then
(()
=>
delay
(
options
.
delay
))
.
then
(()
=>
void
this
.
_checkRequestCount
());
})
.
catch
(
error
=>
{
if
(
retryCount
>=
options
.
retryCount
)
throw
new
Error
(
`Retry give-up for requesting
${
options
.
url
}
!`
,
error
);
debugRequest
(
`Retry requesting
${
options
.
url
}
${
retryCount
+
1
}
times`
);
return
delay
(
options
.
retryDelay
)
.
then
(()
=>
this
.
_removeExists
(
options
))
.
then
(()
=>
this
.
_request
(
options
,
depth
,
retryCount
+
1
));
})
.
catch
(
error
=>
{
debugRequest
(
`Retry give-up for requesting
${
options
.
url
}
after
${
retryCount
}
tries`
);
const
onError
=
options
.
onError
||
noop
;
return
Promise
.
resolve
(
onError
(
error
));
.
catch
(
error
=>
{
if
(
retryCount
>=
options
.
retryCount
)
throw
new
Error
(
`Retry give-up for requesting
${
options
.
url
}
!`
,
error
);
debugRequest
(
`Retry requesting
${
options
.
url
}
${
retryCount
+
1
}
times`
);
return
crawler
.
close
()
.
then
(()
=>
delay
(
options
.
retryDelay
))
.
then
(()
=>
this
.
_removeExists
(
options
))
.
then
(()
=>
this
.
_request
(
options
,
depth
,
retryCount
+
1
));
})
.
catch
(
error
=>
{
debugRequest
(
`Retry give-up for requesting
${
options
.
url
}
after
${
retryCount
}
tries`
);
const
onError
=
options
.
onError
||
noop
;
return
crawler
.
close
()
.
then
(()
=>
Promise
.
resolve
(
onError
(
error
)));
})
));
});
}
Menu
Explore
Projects
Groups
Snippets