Commit | Description |
---|---|
Sat Dec 12 02:06:43 CET 2020 by Michael Peter Christen | updated solr 6.6.6 -> 7.7.3 dropped GSA support (GSA API is still in YaCy Grid) The 6.6.6 solr index works without migration also with 7.7.3 Changed Files: .classpath, build.xml, debian/changelog, defaults/web.xml, htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, lib/commons-collections-3.2.2.jar, lib/lucene-analyzers-common-7.7.3.jar, lib/lucene-analyzers-phonetic-7.7.3.jar, lib/lucene-backward-codecs-7.7.3.jar, lib/lucene-classification-7.7.3.jar, lib/lucene-codecs-7.7.3.jar, lib/lucene-core-7.7.3.jar, lib/lucene-grouping-7.7.3.jar, lib/lucene-highlighter-7.7.3.jar, lib/lucene-join-7.7.3.jar, lib/lucene-memory-7.7.3.jar, lib/lucene-misc-7.7.3.jar, lib/lucene-queries-7.7.3.jar, lib/lucene-queryparser-7.7.3.jar, lib/lucene-spatial-extras-7.7.3.jar, lib/lucene-suggest-7.7.3.jar, lib/noggit-0.8.jar, lib/rrd4j-3.2.jar, lib/solr-core-7.7.3.jar, lib/solr-solrj-7.7.3.jar, lib/zookeeper-3.4.14.jar, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/search/index/Fulltext.java |
Fri Dec 11 00:50:52 CET 2020 by Michael Peter Christen | turned HostBrowser into a admin-only page, now called IndexBrowser This was required because spiders and bots crawled through this page and created load on the peer without use for the user or the YaCy network. Changed Files: defaults/solr.collection.schema, defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/IndexBrowser_p.html, htroot/IndexBrowser_p.java, htroot/IndexBrowser_p.xml, htroot/ViewFile.html, htroot/env/templates/header.template, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, htroot/env/templates/submenuWebStructure.template, htroot/robots.txt, htroot/yacysearchitem.html, htroot/yacysearchitem.java, locales/de.lng, locales/es.lng, locales/fr.lng, locales/it.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/zh.lng, source/net/yacy/search/schema/CollectionSchema.java |
Wed Dec 09 02:36:55 CET 2020 by Michael Peter Christen | Going back to basic authentication for console/shell commands This does not affect security because: - it is going to localhost only - only users who have already access to the pw hash can do this - no clear text pw is transmitted because that is not stored anywhere The switch to basic is required because these commands are required in the context of hosting on root servers and docker containers where a password change must be done. But the password shell command was not working without password which made the concept unusable. This deficit made it virtually impossible for root server operators to use YaCy because they had been unable to set up a proper password. Changed Files: bin/apicall.sh, bin/clearall.sh, bin/clearcache.sh, bin/clearindex.sh, bin/deleteurl.sh, bin/importmediawiki.sh, bin/passwd.sh, source/net/yacy/http/Jetty9YaCySecurityHandler.java, stopYACY.sh |
Thu Dec 03 00:21:15 CET 2020 by Michael Peter Christen | updated jetty from 9.4.17 to 9.4.35 and fixed a bug in ServerSideIncludes that appeared only in that recent version of jetty Changed Files: .classpath, build.xml, lib/jetty-client-9.4.35.v20201120.jar, lib/jetty-continuation-9.4.35.v20201120.jar, lib/jetty-deploy-9.4.35.v20201120.jar, lib/jetty-http-9.4.35.v20201120.jar, lib/jetty-io-9.4.35.v20201120.jar, lib/jetty-jmx-9.4.35.v20201120.jar, lib/jetty-proxy-9.4.35.v20201120.jar, lib/jetty-security-9.4.35.v20201120.jar, lib/jetty-server-9.4.35.v20201120.jar, lib/jetty-servlet-9.4.35.v20201120.jar, lib/jetty-servlets-9.4.35.v20201120.jar, lib/jetty-util-9.4.35.v20201120.jar, lib/jetty-webapp-9.4.35.v20201120.jar, lib/jetty-xml-9.4.35.v20201120.jar, pom.xml, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Tue Jul 28 00:02:33 CEST 2020 by sgaebel | replace org.junit.Assert.assertThat by org.hamcrest.MatcherAssert.assertThat from hamcrest 2.2 to avoid deprecation-warning Changed Files: .classpath, htroot/ConfigParser_p.java, htroot/IndexImportWarc_p.java, htroot/TransNews_p.java, libt/hamcrest-2.2.jar, libt/hamcrest-core-2.2.jar, libt/hamcrest-library-2.2.jar, libt/junit-4.13.jar, pom.xml, source/net/yacy/search/Switchboard.java, test/java/net/yacy/document/ParserTest.java, test/java/net/yacy/document/parser/ooxmlParserTest.java, test/java/net/yacy/document/parser/xlsParserTest.java |
Sun Jul 26 23:44:54 CEST 2020 by sgaebel | removes some warnings: unused imports, params Changed Files: .project, htroot/yacy/profile.java, source/net/yacy/cora/federate/FederateSearchManager.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/protocol/http/auth/YaCyDigestScheme.java, source/net/yacy/cora/storage/Configuration.java, source/net/yacy/cora/util/Html2Image.java, source/net/yacy/document/DateDetection.java, source/net/yacy/document/Parser.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/http/ProxyCacheHandler.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/utils/Gzip.java, source/net/yacy/utils/crypt.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/yacy.java, test/java/net/yacy/peers/NewsPoolTest.java |
Sun Jul 26 17:10:18 CEST 2020 by sgaebel | bump to commons-codec-1.14, commons-compress-1.20, commons-fileupload-1.4, commons-io-2.7, httpclient-4.5.12, httpcore-4.4.13, httpmime-4.5.12 + remove unused commons-jxpath-1.3, htmllexer Changed Files: .classpath, build.xml, lib/commons-codec-1.14.License, lib/commons-codec-1.14.jar, lib/commons-compress-1.20.License, lib/commons-compress-1.20.jar, lib/commons-fileupload-1.4.License, lib/commons-fileupload-1.4.jar, lib/commons-io-2.7.License, lib/commons-io-2.7.jar, lib/httpclient-4.5.12.License, lib/httpclient-4.5.12.jar, lib/httpcore-4.4.13.License, lib/httpcore-4.4.13.jar, lib/httpmime-4.5.12.License, lib/httpmime-4.5.12.jar, libbuild/WebCat-swf/pom.xml, pom.xml, yacy-packages.readme |
Fri Apr 24 11:45:25 CEST 2020 by Michael Peter Christen | replaced json library from JSON.org with libandroid-json-java This fixes https://github.com/yacy/yacy_search_server/issues/347 Changed Files: htroot/Crawler_p.java, htroot/api/snapshot.java, htroot/p2p/seeds.java, htroot/yacy/seedlist.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/document/VocabularyScraper.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/server/serverObjects.java |
Sat Apr 27 21:27:11 CEST 2019 by luccioman | Upgraded Lucene/Solr dependencies from 6.6.5 to 6.6.6 Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-6.6.6.jar, lib/lucene-analyzers-phonetic-6.6.6.jar, lib/lucene-backward-codecs-6.6.6.jar, lib/lucene-classification-6.6.6.jar, lib/lucene-codecs-6.6.6.jar, lib/lucene-core-6.6.6.jar, lib/lucene-grouping-6.6.6.jar, lib/lucene-highlighter-6.6.6.jar, lib/lucene-join-6.6.6.jar, lib/lucene-memory-6.6.6.jar, lib/lucene-misc-6.6.6.jar, lib/lucene-queries-6.6.6.jar, lib/lucene-queryparser-6.6.6.jar, lib/lucene-spatial-extras-6.6.6.jar, lib/lucene-suggest-6.6.6.jar, lib/solr-core-6.6.6.jar, lib/solr-solrj-6.6.6.jar, pom.xml |
Wed Apr 24 09:50:56 CEST 2019 by luccioman | Upgraded Jetty dependencies from 9.4.15.v20190215 to 9.4.17.v20190418 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.17.v20190418.jar, lib/jetty-continuation-9.4.17.v20190418.jar, lib/jetty-deploy-9.4.17.v20190418.jar, lib/jetty-http-9.4.17.v20190418.jar, lib/jetty-io-9.4.17.v20190418.jar, lib/jetty-jmx-9.4.17.v20190418.jar, lib/jetty-proxy-9.4.17.v20190418.jar, lib/jetty-security-9.4.17.v20190418.jar, lib/jetty-server-9.4.17.v20190418.jar, lib/jetty-servlet-9.4.17.v20190418.jar, lib/jetty-servlets-9.4.17.v20190418.jar, lib/jetty-util-9.4.17.v20190418.jar, lib/jetty-webapp-9.4.17.v20190418.jar, lib/jetty-xml-9.4.17.v20190418.jar, pom.xml |
Tue Apr 16 10:25:48 CEST 2019 by luccioman | Upgraded Solr config files with the ones provided by Solr release Fixes #292 Changed Files: defaults/solr/elevate.xml, defaults/solr/lang/contractions_fr.txt, defaults/solr/lang/stopwords_da.txt, defaults/solr/lang/stopwords_de.txt, defaults/solr/lang/stopwords_es.txt, defaults/solr/lang/stopwords_fi.txt, defaults/solr/lang/stopwords_fr.txt, defaults/solr/lang/stopwords_hu.txt, defaults/solr/lang/stopwords_it.txt, defaults/solr/lang/stopwords_nl.txt, defaults/solr/lang/stopwords_no.txt, defaults/solr/lang/stopwords_pt.txt, defaults/solr/lang/stopwords_ru.txt, defaults/solr/lang/stopwords_sv.txt, defaults/solr/lang/userdict_ja.txt, defaults/solr/synonyms.txt |
Sun Apr 14 15:41:13 CEST 2019 by luccioman | Made SNI extension user configurable without the need for server restart TLS Server Name Indication (SNI) extension activation can now be configured with the new Settings_p.html?page=httpClient administration page. SNI extension is also now enabled by default, as in 2019 the unrecognized_name(112) alert is more properly handled by major web servers TLS implementations, following the RFC 6066 standard. Related YaCy issues : #153 #189 and #272 JDK 1.7 bug : https://bugs.java.com/bugdatabase/view_bug.do?bug_id=7127374 Apache httpd issue : https://bz.apache.org/bugzilla/show_bug.cgi?id=56241 RFC 6066 : https://tools.ietf.org/html/rfc6066#section-3 Changed Files: defaults/yacy.init, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_HttpClient.inc, htroot/Settings_p.html, htroot/Settings_p.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, startYACY.bat, startYACY.sh, startYACY_debug.bat |
Tue Apr 09 11:40:21 CEST 2019 by luccioman | Support parsing audio URLs without file extension Added also a Junit for the audio tag parser Changed Files: source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/migration.java, source/net/yacy/search/Switchboard.java, test/java/net/yacy/document/parser/audioTagParserTest.java, test/parsertest/umlaute_windows.aiff, test/parsertest/umlaute_windows.flac, test/parsertest/umlaute_windows.m4a, test/parsertest/umlaute_windows.mp3, test/parsertest/umlaute_windows.ogg, test/parsertest/umlaute_windows.wav |
Tue Apr 02 17:42:50 CEST 2019 by luccioman | Made /yacysearch access rate limitations user configurable With a new admin page at /SearchAccessRate_p.html in menu Network Access > Local Search > Access Rate Limitations Changed Files: defaults/yacy.init, htroot/ConfigPortal_p.java, htroot/ConfigSearchPage_p.java, htroot/SearchAccessRate_p.html, htroot/SearchAccessRate_p.java, htroot/env/templates/submenuAccessTracker.template, htroot/yacysearch.java, source/net/yacy/search/SearchAccessRateConstants.java, source/net/yacy/server/serverSwitch.java |
Mon Mar 18 14:13:10 CET 2019 by luccioman | Upgraded Jetty dependencies from 9.4.14.v20181114 to 9.4.15.v20190215 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.15.v20190215.jar, lib/jetty-continuation-9.4.15.v20190215.jar, lib/jetty-deploy-9.4.15.v20190215.jar, lib/jetty-http-9.4.15.v20190215.jar, lib/jetty-io-9.4.15.v20190215.jar, lib/jetty-jmx-9.4.15.v20190215.jar, lib/jetty-proxy-9.4.15.v20190215.jar, lib/jetty-security-9.4.15.v20190215.jar, lib/jetty-server-9.4.15.v20190215.jar, lib/jetty-servlet-9.4.15.v20190215.jar, lib/jetty-servlets-9.4.15.v20190215.jar, lib/jetty-util-9.4.15.v20190215.jar, lib/jetty-webapp-9.4.15.v20190215.jar, lib/jetty-xml-9.4.15.v20190215.jar, pom.xml |
Thu Feb 21 10:12:39 CET 2019 by luccioman | Upgraded Bootstrap dependency from v3.3.7 to v3.4.1 Non regressions tested on the following platforms : Linux Debian Stretch : - Firefox 60.5.1esr - Chromium 72.0.3626.96 Windows 10 : - Firefox 65.0.1 - Chrome 72.0.3626.109 - Edge 25.10586.672.0 - IE 11.1540.10586.0 Mac OS : - Safari 11.0 Changed Files: htroot/env/bootstrap/css/bootstrap-theme.css, htroot/env/bootstrap/css/bootstrap-theme.css.map, htroot/env/bootstrap/css/bootstrap-theme.min.css, htroot/env/bootstrap/css/bootstrap-theme.min.css.map, htroot/env/bootstrap/css/bootstrap.css, htroot/env/bootstrap/css/bootstrap.css.map, htroot/env/bootstrap/css/bootstrap.min.css, htroot/env/bootstrap/css/bootstrap.min.css.map, htroot/env/bootstrap/js/bootstrap.js, htroot/env/bootstrap/js/bootstrap.min.js, htroot/jslicense.html |
Thu Jan 24 18:43:06 CET 2019 by luccioman | Added the possibility to customize facets sort type and direction Previously search navigators/facets elements were sorted only by counts. Now from the ConfigSearchPage_p.html admin page, sort direction (ascending/descending) and type (on counts or labels) can be customized independently for each navigator. Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, source/net/yacy/cora/federate/FederateSearchManager.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/navigator/FileTypeNavigator.java, source/net/yacy/search/navigator/HostNavigator.java, source/net/yacy/search/navigator/LanguageNavigator.java, source/net/yacy/search/navigator/NameSpaceNavigator.java, source/net/yacy/search/navigator/Navigator.java, source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/navigator/NavigatorSort.java, source/net/yacy/search/navigator/NavigatorSortDirection.java, source/net/yacy/search/navigator/NavigatorSortType.java, source/net/yacy/search/navigator/RestrictedStringNavigator.java, source/net/yacy/search/navigator/StringNavigator.java, source/net/yacy/search/navigator/TokenizedStringNavigator.java, source/net/yacy/search/navigator/YearNavigator.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/server/serverSwitch.java |
Fri Dec 21 15:03:37 CET 2018 by luccioman | Upgraded Jetty dependencies from 9.4.12.v20180830 to 9.4.14.v20181114 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.14.v20181114.jar, lib/jetty-continuation-9.4.14.v20181114.jar, lib/jetty-deploy-9.4.14.v20181114.jar, lib/jetty-http-9.4.14.v20181114.jar, lib/jetty-io-9.4.14.v20181114.jar, lib/jetty-jmx-9.4.14.v20181114.jar, lib/jetty-proxy-9.4.14.v20181114.jar, lib/jetty-security-9.4.14.v20181114.jar, lib/jetty-server-9.4.14.v20181114.jar, lib/jetty-servlet-9.4.14.v20181114.jar, lib/jetty-servlets-9.4.14.v20181114.jar, lib/jetty-util-9.4.14.v20181114.jar, lib/jetty-webapp-9.4.14.v20181114.jar, lib/jetty-xml-9.4.14.v20181114.jar, pom.xml |
Mon Nov 26 18:26:52 CET 2018 by luccioman | Added support for custom DATA path in shell scripts When the YACY_DATA_PATH environment variable is set, shell scripts will now use the given path instead of relative ../DATA which remains the default when the variable is not set. Necessary in the context of Snap package (see issue #254) as YaCy is started with startYACY.sh and an absolute DATA parent path in parameter. Changed Files: bin/apicall.sh, bin/apicat.sh, bin/checkDataFolder.sh, bin/checkalive.sh, bin/checkindex.sh, bin/down.sh, bin/graphicstest.sh, bin/protectedPostApiCall.sh, bin/search.sh, bin/searchall.sh, bin/up.sh |
Thu Oct 25 10:42:12 CEST 2018 by luccioman | Added new crawler attribute for finer control over Media Type detection New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244 Changed Files: htroot/CrawlProfileEditor_p.xml, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/document/TextParser.java, source/net/yacy/search/Switchboard.java |
Tue Sep 25 10:22:50 CEST 2018 by Andreas | Merge pull request #4 from yacy/master Fork update #4 Changed Files: .classpath, CONTRIBUTING.md, README.md, addon/YaCy256x256.png, app.json, build.nsi, build.xml, defaults/yacy.init, defaults/yacy.network.freeworld.unit, defaults/yacy.network.metager.unit, docker/Dockerfile, docker/Dockerfile.alpine, docker/Readme.md, htroot/AccessTracker_p.java, htroot/Blacklist_p.html, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/ConfigAccountList_p.java, htroot/ConfigAccounts_p.html, htroot/ConfigBasic.html, htroot/ConfigBasic.java, htroot/ConfigHeuristics_p.html, htroot/ConfigHeuristics_p.java, htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java, htroot/ConfigParser_p.html, htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java, htroot/ConfigProperties_p.html, htroot/ConfigProperties_p.java, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/Connections_p.html, htroot/CrawlProfileEditor_p.java, htroot/CrawlProfileEditor_p.xml, htroot/CrawlResults.html, htroot/CrawlResults.java, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/CrawlStartScanner_p.java, htroot/CrawlStartSite.html, htroot/Crawler_p.html, htroot/Crawler_p.java, htroot/DictionaryLoader_p.html, htroot/DictionaryLoader_p.java, htroot/Help.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexExport_p.html, htroot/IndexExport_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/IndexImportMediawiki_p.html, htroot/IndexImportMediawiki_p.java, htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, htroot/IndexReIndexMonitor_p.json, htroot/IndexSchema_p.html, htroot/IndexSchema_p.java, htroot/Load_MediawikiWiki.html, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.html, htroot/Load_PHPBB3.java, htroot/Load_RSS_p.html, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Network.java, htroot/PerformanceGraph.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/Performance_p.html, htroot/QuickCrawlLink_p.java, htroot/RankingRWI_p.html, htroot/RankingSolr_p.html, htroot/RegexTest.html, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_Crawler.inc, htroot/Settings_Debug.inc, htroot/Settings_MessageForwarding.inc, htroot/Settings_Proxy.inc, htroot/Settings_ProxyAccess.inc, htroot/Settings_Referrer.inc, htroot/Settings_Seed.inc, htroot/Settings_Seed_UploadFile.inc, htroot/Settings_Seed_UploadFtp.inc, htroot/Settings_Seed_UploadScp.inc, htroot/Settings_ServerAccess.inc, htroot/Settings_UrlProxyAccess.inc, htroot/Settings_p.java, htroot/Status.html, htroot/Status.java, htroot/Surftips.html, htroot/Surftips.java, htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/Translator_p.java, htroot/User.java, htroot/ViewFile.html, htroot/ViewFile.java, htroot/ViewProfile.java, htroot/Vocabulary_p.html, htroot/Vocabulary_p.java, htroot/Wiki.java, htroot/YMarks.html, htroot/api/blacklists/add_entry_p.java, htroot/api/feed.java, htroot/api/push_p.java, htroot/api/snapshot.java, htroot/api/ymarks/get_metadata.java, htroot/api/ynetSearch.java, htroot/env/base.css, htroot/env/templates/header.template, htroot/env/templates/metas.template, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, htroot/goto_p.java, htroot/index.html, htroot/index.java, htroot/js/IndexReIndexMonitor.js, htroot/js/sorttable.js, htroot/js/yacysearch.js, htroot/js/yacysort.js, htroot/jslicense.html, htroot/opensearchdescription.xml, htroot/portalsearch/yacy-portalsearch.html, htroot/portalsearch/yacy-portalsearch.js, htroot/rct_p.java, htroot/sharedBlacklist_p.java, htroot/suggest.java, htroot/terminal_p.html, htroot/yacy/crawlReceipt.java, htroot/yacy/hello.java, htroot/yacy/search.java, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, lib/bcmail-jdk15on-1.60.License, lib/bcmail-jdk15on-1.60.jar, lib/bcpkix-jdk15on-1.60.License, lib/bcpkix-jdk15on-1.60.jar, lib/bcprov-jdk15on-1.60.License, lib/bcprov-jdk15on-1.60.jar, lib/common-image-3.3.2.jar, lib/common-io-3.3.2.jar, lib/common-lang-3.3.2.jar, lib/commons-codec-1.11.License, lib/commons-codec-1.11.jar, lib/commons-compress-1.17.License, lib/commons-compress-1.17.jar, lib/commons-io-2.6.License, lib/commons-io-2.6.jar, lib/fontbox-2.0.11.License, lib/fontbox-2.0.11.jar, lib/httpclient-4.5.5.jar, lib/httpcore-4.4.9.License, lib/httpcore-4.4.9.jar, lib/httpmime-4.5.5.jar, lib/icu4j-62.1.jar, lib/icu4j.License, lib/imageio-bmp-3.3.2.jar, lib/imageio-core-3.3.2.jar, lib/imageio-metadata-3.3.2.jar, lib/imageio-tiff-3.3.2.jar, lib/jaudiotagger-2.2.5.License, lib/jaudiotagger-2.2.5.jar, lib/jcl-over-slf4j-1.7.25.jar, lib/jetty-client-9.4.12.v20180830.jar, lib/jetty-continuation-9.4.12.v20180830.jar, lib/jetty-deploy-9.4.12.v20180830.jar, lib/jetty-http-9.4.12.v20180830.jar, lib/jetty-io-9.4.12.v20180830.jar, lib/jetty-jmx-9.4.12.v20180830.jar, lib/jetty-proxy-9.4.12.v20180830.jar, lib/jetty-security-9.4.12.v20180830.jar, lib/jetty-server-9.4.12.v20180830.jar, lib/jetty-servlet-9.4.12.v20180830.jar, lib/jetty-servlets-9.4.12.v20180830.jar, lib/jetty-util-9.4.12.v20180830.jar, lib/jetty-webapp-9.4.12.v20180830.jar, lib/jetty-xml-9.4.12.v20180830.jar, lib/jsoup-1.11.3.jar, lib/jwat-archive-common-1.1.1.jar, lib/jwat-common-1.1.1.jar, lib/jwat-gzip-1.1.1.jar, lib/jwat-warc-1.1.1.jar, lib/log4j-over-slf4j-1.7.25.jar, lib/lucene-analyzers-common-6.6.5.jar, lib/lucene-analyzers-phonetic-6.6.5.jar, lib/lucene-backward-codecs-6.6.5.jar, lib/lucene-classification-6.6.5.jar, lib/lucene-codecs-6.6.5.jar, lib/lucene-core-6.6.5.jar, lib/lucene-grouping-6.6.5.jar, lib/lucene-highlighter-6.6.5.jar, lib/lucene-join-6.6.5.jar, lib/lucene-memory-6.6.5.jar, lib/lucene-misc-6.6.5.jar, lib/lucene-queries-6.6.5.jar, lib/lucene-queryparser-6.6.5.jar, lib/lucene-spatial-extras-6.6.5.jar, lib/lucene-suggest-6.6.5.jar, lib/metadata-extractor-2.11.0.License, lib/metadata-extractor-2.11.0.jar, lib/pdfbox-2.0.11.License, lib/pdfbox-2.0.11.jar, lib/poi-3.17.License, lib/poi-3.17.jar, lib/poi-scratchpad-3.17.jar, lib/slf4j-api-1.7.25.jar, lib/slf4j-jdk14-1.7.25.jar, lib/solr-core-6.6.5.jar, lib/solr-solrj-6.6.5.jar, lib/xz-1.8.License, lib/xz-1.8.jar, libbuild/GitRevMavenTask/.gitignore, libbuild/GitRevMavenTask/src/GitRevMavenTask.java, libbuild/J7Zip-modified/.gitignore, locales/de.lng, locales/el.lng, locales/fr.lng, locales/hi.lng, locales/it.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/sk.lng, locales/uk.lng, locales/zh.lng, pom.xml, snap/snapcraft.yaml, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/document/analysis/Classification.java, source/net/yacy/cora/document/feed/Hit.java, source/net/yacy/cora/document/feed/RSSMessage.java, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/SolrFederateSearchConnector.java, source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/instance/ResponseAccumulator.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/instance/ShardInstance.java, source/net/yacy/cora/federate/solr/responsewriter/EmbeddedSolrResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/SolrjResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/order/Base64Order.java, source/net/yacy/cora/order/Digest.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/StrictSizeLimitEntityWrapper.java, source/net/yacy/cora/protocol/http/StrictSizeLimitResponseInterceptor.java, source/net/yacy/cora/protocol/tld/GenericTLD.java, source/net/yacy/cora/protocol/tld/InternationalizedCountryCodeTLD.java, source/net/yacy/cora/protocol/tld/SponsoredTLD.java, source/net/yacy/cora/sorting/Array.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/sorting/ConcurrentScoreMap.java, source/net/yacy/cora/sorting/OrderedScoreMap.java, source/net/yacy/cora/sorting/ScoreMap.java, source/net/yacy/cora/sorting/ScoreMapUpdatesListener.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/cora/util/ByteBuffer.java, source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/cora/util/Html2Image.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/CrawlStarterFromScraper.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/FileCrawlStarterTask.java, source/net/yacy/crawler/HostBalancer.java, source/net/yacy/crawler/HostQueue.java, source/net/yacy/crawler/RecrawlBusyThread.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/crawler/data/NoticedURL.java, source/net/yacy/crawler/data/Snapshots.java, source/net/yacy/crawler/data/Transactions.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/RSSLoader.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/StreamResponse.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/DidYouMean.java, source/net/yacy/data/TransactionManager.java, source/net/yacy/data/Translator.java, source/net/yacy/data/UserDB.java, source/net/yacy/data/WorkTables.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/DateDetection.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/Parser.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/Tokenizer.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/WarcImporter.java, source/net/yacy/document/parser/AbstractCompressorParser.java, source/net/yacy/document/parser/XZParser.java, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/document/parser/html/AbstractScraper.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/Scraper.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/svgParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java, source/net/yacy/document/parser/rssParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/gui/YaCyApp.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/http/InetPathAccessHandler.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/http/Jetty9YaCySecurityHandler.java, source/net/yacy/http/MonitorHandler.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/http/servlets/UrlProxyServlet.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/http/servlets/YaCyQoSFilter.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/kelondro/workflow/WorkflowTask.java, source/net/yacy/migration.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/Transmission.java, source/net/yacy/peers/graphics/EncodedImage.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/ProfilingGraph.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/BlacklistHelper.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/AutoSearch.java, source/net/yacy/search/MemoryTracker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/ErrorCache.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SingleDocumentMatcher.java, source/net/yacy/search/navigator/LanguageNavigator.java, source/net/yacy/search/navigator/Navigator.java, source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/navigator/StringNavigator.java, source/net/yacy/search/navigator/TokenizedStringNavigator.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SecondarySearchSuperviser.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/snippet/TextSnippetStatistics.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/serverObjects.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/utils/translation/GenerateMasterXliff.java, source/net/yacy/utils/translation/TranslationManager.java, source/net/yacy/visualization/ChartPlotter.java, source/net/yacy/visualization/ImageViewer.java, source/net/yacy/yacy.java, startYACY.sh, stopYACY.sh, test/Vocabulary_pTest.java, test/java/net/yacy/cora/date/GenericFormatterTest.java, test/java/net/yacy/cora/document/analysis/ClassificationTest.java, test/java/net/yacy/cora/document/id/DigestURLHashPerfTest.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java, test/java/net/yacy/cora/language/synonyms/AutotaggingLibraryTest.java, test/java/net/yacy/cora/protocol/DomainsTest.java, test/java/net/yacy/cora/protocol/HeaderFrameworkTest.java, test/java/net/yacy/cora/protocol/ResponseHeaderTest.java, test/java/net/yacy/cora/util/Html2ImageTest.java, test/java/net/yacy/crawler/HostBalancerTest.java, test/java/net/yacy/dbtest.java, test/java/net/yacy/document/DateDetectionTest.java, test/java/net/yacy/document/TextParserTest.java, test/java/net/yacy/document/TokenizerTest.java, test/java/net/yacy/document/parser/XZParserTest.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java, test/java/net/yacy/document/parser/htmlParserTest.java, test/java/net/yacy/document/parser/vsdParserTest.java, test/java/net/yacy/http/InetPathAccessHandlerTest.java, test/java/net/yacy/kelondro/io/RecordsTest.java, test/java/net/yacy/migrationTest.java, test/java/net/yacy/peers/graphics/ProfilingGraphTest.java, test/java/net/yacy/repository/BlacklistTest.java, test/java/net/yacy/search/index/SegmentTest.java, test/java/net/yacy/search/index/SingleDocumentMatcherTest.java, test/java/net/yacy/search/query/QueryParamsTest.java, test/java/net/yacy/search/snippet/TextSnippetTest.java, test/parsertest/umlaute_html_utf8.html.xz, test/parsertest/umlaute_html_xml_txt_gnu.txz, test/parsertest/umlaute_linux.txt.xz |
Sat Sep 22 14:40:18 CEST 2018 by luccioman | Upgraded Solr and Lucene dependencies from 6.6.3 to 6.6.5 Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-6.6.5.jar, lib/lucene-analyzers-phonetic-6.6.5.jar, lib/lucene-backward-codecs-6.6.5.jar, lib/lucene-classification-6.6.5.jar, lib/lucene-codecs-6.6.5.jar, lib/lucene-core-6.6.5.jar, lib/lucene-grouping-6.6.5.jar, lib/lucene-highlighter-6.6.5.jar, lib/lucene-join-6.6.5.jar, lib/lucene-memory-6.6.5.jar, lib/lucene-misc-6.6.5.jar, lib/lucene-queries-6.6.5.jar, lib/lucene-queryparser-6.6.5.jar, lib/lucene-spatial-extras-6.6.5.jar, lib/lucene-suggest-6.6.5.jar, lib/solr-core-6.6.5.jar, lib/solr-solrj-6.6.5.jar, pom.xml |
Wed Sep 19 09:18:36 CEST 2018 by luccioman | Enforced access controls to System settings pages Changed Files: htroot/SettingsAck_p.java, htroot/Settings_Crawler.inc, htroot/Settings_Debug.inc, htroot/Settings_MessageForwarding.inc, htroot/Settings_Proxy.inc, htroot/Settings_ProxyAccess.inc, htroot/Settings_Referrer.inc, htroot/Settings_Seed.inc, htroot/Settings_Seed_UploadFile.inc, htroot/Settings_Seed_UploadFtp.inc, htroot/Settings_Seed_UploadScp.inc, htroot/Settings_ServerAccess.inc, htroot/Settings_UrlProxyAccess.inc, htroot/Settings_p.java |
Fri Sep 14 14:03:44 CEST 2018 by luccioman | Upgraded Jetty dependencies from 9.4.11.v20180605 to 9.4.12.v20180830 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.12.v20180830.jar, lib/jetty-continuation-9.4.12.v20180830.jar, lib/jetty-deploy-9.4.12.v20180830.jar, lib/jetty-http-9.4.12.v20180830.jar, lib/jetty-io-9.4.12.v20180830.jar, lib/jetty-jmx-9.4.12.v20180830.jar, lib/jetty-proxy-9.4.12.v20180830.jar, lib/jetty-security-9.4.12.v20180830.jar, lib/jetty-server-9.4.12.v20180830.jar, lib/jetty-servlet-9.4.12.v20180830.jar, lib/jetty-servlets-9.4.12.v20180830.jar, lib/jetty-util-9.4.12.v20180830.jar, lib/jetty-webapp-9.4.12.v20180830.jar, lib/jetty-xml-9.4.12.v20180830.jar, pom.xml |
Thu Aug 16 18:23:42 CEST 2018 by luccioman | Updated pdf cache clear steps consistently with current pdfbox version - Removed calls to no more existing clearResources functions (on PDFont class and its children) since upgrade to pdfbox 2.n.n - Removed hacky usage of protected internal ClassLoader function. This removes the warnings displayed when running with JDK9 or JDK10 : [java] WARNING: Illegal reflective access by net.yacy.document.parser.pdfParser$ResourceCleaner (file:<path>) to method java.lang.ClassLoader.findLoadedClass(java.lang.String) [java] WARNING: Please consider reporting this to the maintainers of net.yacy.document.parser.pdfParser$ResourceCleaner [java] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations [java] WARNING: All illegal access operations will be denied in a future release Crawling thousands of pdf documents from various sources after modifications applied, revealed no new memory leak related to pdfbox (measurements done with JVisualVM). Changed Files: source/net/yacy/document/parser/pdfParser.java, source/net/yacy/search/Switchboard.java |
Wed Aug 15 10:07:39 CEST 2018 by luccioman | Added a parser for XZ compressed archives. As suggested by LA_FORGE on mantis 781 (http://mantis.tokeek.de/view.php?id=781) Changed Files: .classpath, build.xml, lib/xz-1.8.License, lib/xz-1.8.jar, pom.xml, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/AbstractCompressorParser.java, source/net/yacy/document/parser/XZParser.java, test/java/net/yacy/document/parser/XZParserTest.java, test/parsertest/umlaute_html_utf8.html.xz, test/parsertest/umlaute_html_xml_txt_gnu.txz, test/parsertest/umlaute_linux.txt.xz |
Fri Jul 06 11:30:30 CEST 2018 by luccioman | Added and updated hint messages about remote crawler status To help identify why remote crawl results may not be received. Changed Files: htroot/CrawlResults.html, htroot/CrawlResults.java, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.html, htroot/Crawler_p.java, locales/de.lng, locales/hi.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Wed Jul 04 10:28:32 CEST 2018 by luccioman | Added server side columns sorting on the Process Scheduler table For easier usage of large tables in the Table_API_p.html page. Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/env/base.css, locales/de.lng, locales/it.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/search/Switchboard.java |
Thu Jun 28 14:59:23 CEST 2018 by luccioman | Fixed unsafe conccurent access to generic SimpleDateFormat instances SimpleDateFormat must not be used by concurrent threads without synchronization for parsing or formating dates as it is not thread-safe (internally holds a calendar instance that is not synchronized). Prefer now DateTimeFormatter when possible as it is thread-safe without concurrent access performance bottleneck (does not internally use synchronization locks). Changed Files: htroot/ConfigAccountList_p.java, htroot/IndexImportMediawiki_p.java, htroot/Table_API_p.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/document/DateDetection.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/search/index/Fulltext.java, test/java/net/yacy/cora/date/GenericFormatterTest.java |
Wed Jun 20 09:33:26 CEST 2018 by luccioman | Upgraded Jetty dependency from 9.4.9 to 9.4.11 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.11.v20180605.jar, lib/jetty-continuation-9.4.11.v20180605.jar, lib/jetty-deploy-9.4.11.v20180605.jar, lib/jetty-http-9.4.11.v20180605.jar, lib/jetty-io-9.4.11.v20180605.jar, lib/jetty-jmx-9.4.11.v20180605.jar, lib/jetty-proxy-9.4.11.v20180605.jar, lib/jetty-security-9.4.11.v20180605.jar, lib/jetty-server-9.4.11.v20180605.jar, lib/jetty-servlet-9.4.11.v20180605.jar, lib/jetty-servlets-9.4.11.v20180605.jar, lib/jetty-util-9.4.11.v20180605.jar, lib/jetty-webapp-9.4.11.v20180605.jar, lib/jetty-xml-9.4.11.v20180605.jar, pom.xml, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Tue Jun 19 10:12:20 CEST 2018 by luccioman | Added a new crawler document filter type using Solr syntax This makes possbile to set up much more advanced document crawl filters, by filtering on one or more document indexed fields before inserting in the index. Changed Files: htroot/CrawlProfileEditor_p.xml, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.html, htroot/Crawler_p.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SingleDocumentMatcher.java, test/java/net/yacy/search/index/SingleDocumentMatcherTest.java |
Wed May 23 14:45:35 CEST 2018 by luccioman | Small perf improvement : initialize threads names early when possible Initializing Thread names using the Thread constructor parameter is faster as it already sets a thread name even if no customized one is given, while an additional call to the Thread.setName() function internally do synchronized access, eventually runs access check on the security manager and performs a native call. Profiling a running YaCy server revealed that the total processing time spent on Thread.setName() for a typical p2p search was in the range of seconds. Changed Files: source/net/yacy/cora/federate/opensearch/SRURSSConnector.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/MirrorSolrConnector.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/sorting/Array.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/crawler/data/Transactions.java, source/net/yacy/crawler/robots/RobotsTxt.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/gui/YaCyApp.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/search/MemoryTracker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SecondarySearchSuperviser.java |
Tue May 22 17:46:12 CEST 2018 by luccioman | Updated YaCy home page embedded links from http to https scheme Changed Files: app.json, defaults/yacy.init, htroot/ConfigPortal_p.java, htroot/Help.html, htroot/api/feed.java, htroot/env/templates/header.template, htroot/env/templates/metas.template, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, htroot/opensearchdescription.xml, htroot/terminal_p.html |
Thu Apr 26 09:39:30 CEST 2018 by luccioman | Removed some remaining uses of deprecated Seed.getIP() Changed Files: htroot/Load_MediawikiWiki.html, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.html, htroot/Load_PHPBB3.java, htroot/Network.java, htroot/Status.java, htroot/Surftips.java, htroot/api/ynetSearch.java, htroot/goto_p.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Sun Apr 15 09:55:08 CEST 2018 by luccioman | Added analysis optional setting to compute statistics on text snippets Thus producing some basic stats on processing times for snippets generation and counts on snippets per source type. Changed Files: defaults/yacy.init, htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java, htroot/SettingsAck_p.java, htroot/Settings_Debug.inc, htroot/Settings_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/snippet/TextSnippetStatistics.java |
Sat Apr 07 23:39:03 CEST 2018 by reger | upd to jetty-9.4.9.v20180320 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.9.v20180320.jar, lib/jetty-continuation-9.4.9.v20180320.jar, lib/jetty-deploy-9.4.9.v20180320.jar, lib/jetty-http-9.4.9.v20180320.jar, lib/jetty-io-9.4.9.v20180320.jar, lib/jetty-jmx-9.4.9.v20180320.jar, lib/jetty-proxy-9.4.9.v20180320.jar, lib/jetty-security-9.4.9.v20180320.jar, lib/jetty-server-9.4.9.v20180320.jar, lib/jetty-servlet-9.4.9.v20180320.jar, lib/jetty-servlets-9.4.9.v20180320.jar, lib/jetty-util-9.4.9.v20180320.jar, lib/jetty-webapp-9.4.9.v20180320.jar, lib/jetty-xml-9.4.9.v20180320.jar, pom.xml |
Fri Apr 06 15:16:54 CEST 2018 by luccioman | Adjusted Solr select servlet output when using an external Solr only - Use the EnhancedXMLResponseWriter only when requested output is "exml" - Use the Standard Solr writers when possible, for example for json, xml or javabin output formats - Return an error when the requested format can not been rendered with an external Solr server only Important : this modification is necessary for peers using exclusively an external Solr server to be reachable as robinson targets in p2p search, as the binary format ("javabin") is the default Solr exchange format for peers. Before this, when a peer requested a remote one attached only to an external Solr (no embedded one), it ended with "Invalid type" error, as the remote peer answered with xml although binary format was requested. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EmbeddedSolrResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/search/query/AccessTracker.java |
Thu Apr 05 18:15:45 CEST 2018 by luccioman | Upgraded Solr dependencies from 6.6.2 to 6.6.3 Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-6.6.3.jar, lib/lucene-analyzers-phonetic-6.6.3.jar, lib/lucene-backward-codecs-6.6.3.jar, lib/lucene-classification-6.6.3.jar, lib/lucene-codecs-6.6.3.jar, lib/lucene-core-6.6.3.jar, lib/lucene-grouping-6.6.3.jar, lib/lucene-highlighter-6.6.3.jar, lib/lucene-join-6.6.3.jar, lib/lucene-memory-6.6.3.jar, lib/lucene-misc-6.6.3.jar, lib/lucene-queries-6.6.3.jar, lib/lucene-queryparser-6.6.3.jar, lib/lucene-spatial-extras-6.6.3.jar, lib/lucene-suggest-6.6.3.jar, lib/solr-core-6.6.3.jar, lib/solr-solrj-6.6.3.jar, pom.xml |
Wed Apr 04 18:16:26 CEST 2018 by luccioman | Optionally allow external Solr server with self-signed certificate This is necessary when you want to attach to a dedicated external Solr server protected with basic http authentication and requested over https but having only a self-signed certificate. Changed Files: defaults/yacy.init, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, source/net/yacy/cora/federate/SolrFederateSearchConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/AutoSearch.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Apr 02 15:04:56 CEST 2018 by luccioman | Adjusted localhost/qualified account admin access informational texts. Following remarks from @etam on issue #170 Changed Files: htroot/ConfigAccounts_p.html, locales/de.lng, locales/fr.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Fri Feb 16 11:35:15 CET 2018 by luccioman | Fixed CrawlStartExpert.html HTML validation errors Validated with Nu Html Checker 17.11.1 Changed Files: htroot/CrawlStartExpert.html, locales/de.lng, locales/fr.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/sk.lng, locales/uk.lng, locales/zh.lng |
Wed Feb 14 07:51:07 CET 2018 by luccioman | Adjusted last blacklist entry example for a more accurate description As discussed in issue #160 , blacklist entries can indeed currently not be "complete" regular expressions, but must be structured as a domain part, a separator character ('/'), and a path part. Changed Files: htroot/Blacklist_p.html, locales/de.lng, locales/fr.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Tue Feb 06 10:25:38 CET 2018 by luccioman | Added basic support for autotagging microdata annotated item types. With the appropriate vocabulary settings in Vocabulary_p.html page, this can produce Vocabulary search facets displaying item types referenced in html documents by microdata annotation. Tested notably, but not limited to, vocabulary classes/types defined by Schema.org and Dublin Core. Changed Files: defaults/yacy.init, htroot/Vocabulary_p.html, htroot/Vocabulary_p.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/language/synonyms/AutotaggingLibrary.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, test/java/net/yacy/cora/language/synonyms/AutotaggingLibraryTest.java |
Sat Dec 23 18:56:17 CET 2017 by luccioman | Added optional search parameter/setting to control content domain filter Thus allowing to choose at configuration or per search request, whether extending or not results beyond strict content domain filter (image, video, audio or application). Related graphical controls to be added to user interface. Changed Files: defaults/yacy.init, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Tue Dec 19 13:52:05 CET 2017 by luccioman | Do locale independant case conversion on hosts, schemes, and file exts. Required for proper operation when the default system locale is Turkish, as dottless and dotted i characters have specific case conversion rules in this language. Changed Files: htroot/api/ymarks/get_metadata.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/robots/RobotsTxtEntry.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Fri Dec 15 11:28:46 CET 2017 by luccioman | Started implementing optional https preference for protocol operations Introduced through the new configurable setting network.unit.protocol.https.preferred, defaulting to false for now. Let choose to prefer using https when available on remote peers to perform YaCy protocol operations including notably hello or transferRWI. Not yet implemented for every YaCy protocol operations. Changed Files: defaults/yacy.init, htroot/MessageSend_p.java, htroot/Network.java, htroot/yacy/hello.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/search/SwitchboardConstants.java |
Sat Dec 09 22:29:35 CET 2017 by Michael Peter Christen | added a crawl filter based on <div> tag class names When a crawl is started, a new field to exclude content from scraping is available. The field can be identified with the class name of div tags. All text contained in such a div tag where the configured class name(s) match are not indexed, while the remaining page is indexed. Changed Files: htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.java, htroot/QuickCrawlLink_p.java, source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/data/ymark/YMarkCrawlStart.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Fri Dec 08 15:12:08 CET 2017 by luccioman | Removed use of deprecated Jetty IPAccessHandler for client filtering. Upgraded to InetAccessHandler. Added InetPathAccessHandler extension to InetAccessHandler to maintain path patterns capability previously available in IPAccessHandler but lost in InetAccessHandler. Filtering on IPv6 addresses is now supported. Support for deprecated pattern formats such as "192.168." and "192.168.1.1/path" has been removed, but startup automated migration should convert such patterns eventually present in serverClient. Changed Files: defaults/yacy.init, htroot/SettingsAck_p.java, htroot/Settings_ServerAccess.inc, locales/ru.lng, source/net/yacy/http/InetPathAccessHandler.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/migration.java, source/net/yacy/yacy.java, test/java/net/yacy/http/InetPathAccessHandlerTest.java, test/java/net/yacy/migrationTest.java |
Thu Dec 07 00:24:33 CET 2017 by reger | upd to Jetty-9.4.8.v20171121 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.8.v20171121.jar, lib/jetty-continuation-9.4.8.v20171121.jar, lib/jetty-deploy-9.4.8.v20171121.jar, lib/jetty-http-9.4.8.v20171121.jar, lib/jetty-io-9.4.8.v20171121.jar, lib/jetty-jmx-9.4.8.v20171121.jar, lib/jetty-proxy-9.4.8.v20171121.jar, lib/jetty-security-9.4.8.v20171121.jar, lib/jetty-server-9.4.8.v20171121.jar, lib/jetty-servlet-9.4.8.v20171121.jar, lib/jetty-servlets-9.4.8.v20171121.jar, lib/jetty-util-9.4.8.v20171121.jar, lib/jetty-webapp-9.4.8.v20171121.jar, lib/jetty-xml-9.4.8.v20171121.jar, pom.xml |
Mon Dec 04 08:48:37 CET 2017 by luccioman | Use HTTP Post operation for resetting memory monitoring state. Fixes issue #145 Also added textual hint on the button, and display it only when it makes sense, that is to say when the memory state is 'exhausted'. Changed Files: htroot/PerformanceQueues_p.java, htroot/Performance_p.html, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Fri Nov 24 14:10:41 CET 2017 by luccioman | Made possible to use https for remote search on peers with SSL enabled. Default is still http to prevent any regressions, but a new setting is available to choose https as the preferred protocol to perform remote searches. New configuration setting 'remotesearch.https.preferred' is manually editable in yacy.conf file or in Advanced Properties page (/ConfigProperties_p.html). Should be enabled as default in the future for improved privacy. Https could also eventually be used for other peers communications. Changed Files: defaults/yacy.init, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Seed.java, source/net/yacy/search/SwitchboardConstants.java |
Thu Nov 09 09:30:20 CET 2017 by luccioman | Upgraded com.twelvemonkeys.imageio dependencies from 3.3.1 to 3.3.2 Changed Files: .classpath, build.xml, lib/common-image-3.3.2.jar, lib/common-io-3.3.2.jar, lib/common-lang-3.3.2.jar, lib/imageio-bmp-3.3.2.jar, lib/imageio-core-3.3.2.jar, lib/imageio-metadata-3.3.2.jar, lib/imageio-tiff-3.3.2.jar, pom.xml |
Thu Oct 26 07:51:18 CEST 2017 by luccioman | Enable HTTP Digest authentication for non admin users. Also ensure authentication is not lost by Digest timeout when navigating between index.html and search results page. This way, running searches with extended features on a remote peer or a password protected peer works with a regular user (with "Extended search" rights). When authenticating on the search page with a user without "Extended search" rights, it appears as authenticated, but has just its usual access to the public search features. Changed Files: htroot/Blog.java, htroot/BlogComments.java, htroot/User.java, htroot/env/templates/header.template, htroot/env/templates/simpleSearchHeader.template, htroot/index.html, htroot/index.java, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchtrailer.java, source/net/yacy/data/UserDB.java, source/net/yacy/search/Switchboard.java |
Sun Oct 22 20:00:00 CEST 2017 by reger | upd to Solr 6.6.2 Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-6.6.2.jar, lib/lucene-analyzers-phonetic-6.6.2.jar, lib/lucene-backward-codecs-6.6.2.jar, lib/lucene-classification-6.6.2.jar, lib/lucene-codecs-6.6.2.jar, lib/lucene-core-6.6.2.jar, lib/lucene-grouping-6.6.2.jar, lib/lucene-highlighter-6.6.2.jar, lib/lucene-join-6.6.2.jar, lib/lucene-memory-6.6.2.jar, lib/lucene-misc-6.6.2.jar, lib/lucene-queries-6.6.2.jar, lib/lucene-queryparser-6.6.2.jar, lib/lucene-spatial-extras-6.6.2.jar, lib/lucene-suggest-6.6.2.jar, lib/solr-core-6.6.2.jar, lib/solr-solrj-6.6.2.jar, pom.xml |
Sat Oct 21 10:57:36 CEST 2017 by luccioman | Added an optional login link/status to the search public top nav bar. Thus allowing a more convenient way (wihout the need to go to the admin section) to login when searching on your remote or password protected peer and benefit from extended search features such as Heuristics, Bookmarking or JavasScript resorting. Can be disabled using the ConfigSearchPage_p.html. Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/env/templates/simpleSearchHeader.template, htroot/yacysearch.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Thu Oct 12 07:16:19 CEST 2017 by luccioman | Reduced number of search navigators refresh requests in JS resort mode The SearchEvent listen to changes on each of its navigators, and the information about their overall state is sent with each fetched search item (as a "data-nav-generation" attribute). Then the browser can regularly fetch a fresh version of yacysearchtrailer.html only if necessary (when that nav-generation value change). Changed Files: htroot/js/yacysort.js, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/sorting/ConcurrentScoreMap.java, source/net/yacy/cora/sorting/OrderedScoreMap.java, source/net/yacy/cora/sorting/ScoreMap.java, source/net/yacy/cora/sorting/ScoreMapUpdatesListener.java, source/net/yacy/search/navigator/Navigator.java, source/net/yacy/search/navigator/StringNavigator.java, source/net/yacy/search/query/SearchEvent.java |
Mon Oct 09 14:13:46 CEST 2017 by luccioman | Add a configurable limit to tags initially displayed in search results When the limit is reached, a button allow expanding/collapsing remaining tags. When this feature is activated without a limit to the number of displayed tags, when encountering search results with a very large number of keywords, the results page can become almost unusable (very long vertical scrollbar) Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/env/base.css, htroot/js/yacysearch.js, htroot/yacysearchitem.html, htroot/yacysearchitem.java, source/net/yacy/search/SwitchboardConstants.java |
Sat Oct 07 12:29:55 CEST 2017 by Andreas | Merge pull request #3 from yacy/master Fork update Changed Files: .classpath, .travis.yml, build.xml, debian/control, htroot/ConfigBasic.html, htroot/ConfigBasic.java, htroot/ConfigSearchPage_p.html, lib/jetty-client-9.4.7.v20170914.jar, lib/jetty-continuation-9.4.7.v20170914.jar, lib/jetty-deploy-9.4.7.v20170914.jar, lib/jetty-http-9.4.7.v20170914.jar, lib/jetty-io-9.4.7.v20170914.jar, lib/jetty-jmx-9.4.7.v20170914.jar, lib/jetty-proxy-9.4.7.v20170914.jar, lib/jetty-security-9.4.7.v20170914.jar, lib/jetty-server-9.4.7.v20170914.jar, lib/jetty-servlet-9.4.7.v20170914.jar, lib/jetty-servlets-9.4.7.v20170914.jar, lib/jetty-util-9.4.7.v20170914.jar, lib/jetty-webapp-9.4.7.v20170914.jar, lib/jetty-xml-9.4.7.v20170914.jar, pom.xml, source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/cora/util/Html2Image.java, source/net/yacy/data/Translator.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/search/schema/WebgraphConfiguration.java, test/java/net/yacy/document/parser/bzipParserTest.java, test/java/net/yacy/kelondro/rwi/ReferenceContainerTest.java, test/parsertest/umlaute_html_utf8.html.bz2, test/parsertest/umlaute_html_xml_txt_gnu.tbz2, test/parsertest/umlaute_linux.txt.bz2 |
Mon Oct 02 00:50:30 CEST 2017 by reger | upd to Jetty-9.4.7.v20170914 Changed Files: .classpath, build.xml, lib/jetty-client-9.4.7.v20170914.jar, lib/jetty-continuation-9.4.7.v20170914.jar, lib/jetty-deploy-9.4.7.v20170914.jar, lib/jetty-http-9.4.7.v20170914.jar, lib/jetty-io-9.4.7.v20170914.jar, lib/jetty-jmx-9.4.7.v20170914.jar, lib/jetty-proxy-9.4.7.v20170914.jar, lib/jetty-security-9.4.7.v20170914.jar, lib/jetty-server-9.4.7.v20170914.jar, lib/jetty-servlet-9.4.7.v20170914.jar, lib/jetty-servlets-9.4.7.v20170914.jar, lib/jetty-util-9.4.7.v20170914.jar, lib/jetty-webapp-9.4.7.v20170914.jar, lib/jetty-xml-9.4.7.v20170914.jar, pom.xml |
Fri Sep 29 23:22:39 CEST 2017 by Andreas | Merge pull request #2 from yacy/master Merge #2 Changed Files: htroot/env/templates/header.template, htroot/index.html, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchtrailer.java, locales/de.lng, locales/fr.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java |
Fri Sep 29 19:18:12 CEST 2017 by luccioman | Ensure private search features are not lost on Digest auth timeout This is a fix for mantis 766 ( http://mantis.tokeek.de/view.php?id=766 ) Since the upgrade to Digest authentication, access to protected search features was indeed disabled once the Digest nonce timed out. After Digest auth timeout the browser no more sent authentication information and as the search results page is not private, protected features were simply be hidden without asking browser again for authentication. Adding a supplementary parameter when accessing the search results as authenticated fixes this. Changed Files: htroot/env/templates/header.template, htroot/index.html, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryParams.java |
Wed Sep 27 23:32:00 CEST 2017 by Andreas | Merge pull request #1 from yacy/master update Changed Files: .classpath, build.xml, defaults/yacy.init, htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/HostBrowser.html, htroot/IndexControlRWIs_p.java, htroot/env/templates/footer.template, htroot/env/templates/submenuDesign.template, htroot/env/yacysort.css, htroot/js/accessibleHistogram.js, htroot/js/raphael.min.js, htroot/js/yacysearch.js, htroot/js/yacysort.js, htroot/jslicense.html, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, lib/jsonic-1.3.10.jar, lib/lucene-analyzers-common-6.6.1.jar, lib/lucene-analyzers-phonetic-6.6.1.jar, lib/lucene-backward-codecs-6.6.1.jar, lib/lucene-classification-6.6.1.jar, lib/lucene-codecs-6.6.1.jar, lib/lucene-core-6.6.1.jar, lib/lucene-grouping-6.6.1.jar, lib/lucene-highlighter-6.6.1.jar, lib/lucene-join-6.6.1.jar, lib/lucene-memory-6.6.1.jar, lib/lucene-misc-6.6.1.jar, lib/lucene-queries-6.6.1.jar, lib/lucene-queryparser-6.6.1.jar, lib/lucene-spatial-extras-6.6.1.jar, lib/lucene-suggest-6.6.1.jar, lib/solr-core-6.6.1.jar, lib/solr-solrj-6.6.1.jar, pom.xml, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/QueryParams.java |
Mon Sep 25 09:19:08 CEST 2017 by luccioman | Made the dates navigator max elements number user configurable. Also used object properties on QueryParams instances, rather than using mutable class (static) properties. Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchtrailer.java, source/net/yacy/cora/federate/FederateSearchManager.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/QueryParams.java |
Sun Sep 17 08:25:14 CEST 2017 by reger | update jars for upd solr 6.6. commit for ant Changed Files: lib/jsonic-1.3.10.jar, lib/lucene-analyzers-common-6.6.1.jar, lib/lucene-analyzers-phonetic-6.6.1.jar, lib/lucene-backward-codecs-6.6.1.jar, lib/lucene-classification-6.6.1.jar, lib/lucene-codecs-6.6.1.jar, lib/lucene-core-6.6.1.jar, lib/lucene-grouping-6.6.1.jar, lib/lucene-highlighter-6.6.1.jar, lib/lucene-join-6.6.1.jar, lib/lucene-memory-6.6.1.jar, lib/lucene-misc-6.6.1.jar, lib/lucene-queries-6.6.1.jar, lib/lucene-queryparser-6.6.1.jar, lib/lucene-spatial-extras-6.6.1.jar, lib/lucene-suggest-6.6.1.jar, lib/solr-core-6.6.1.jar, lib/solr-solrj-6.6.1.jar |
Wed Sep 06 16:58:40 CEST 2017 by luccioman | Improved search navigators counters accuracy and consistency. - added some missing increments from RWI results - decrement relevant navigator counts when solr or RWI results are evicted because duplicates detection or constraints checked belatedly - do not compute facets when unnecessary to avoid unwanted CPU load - do not increment from facets when already done - do not rely on facets on remote solr peers requests, as most of the time only a limited part of their total results if fetched (thus also preventing unnecessary load on remote peers) - use a concurrency friendly score map for the dates navigators to prevent unwanted ConcurrentModificationExceptions This improves the situation for the most obvious inconsistencies in search navigators counts, but more has to be done for a true accuracy (notably when query modifiers constraints are applied belatedly - after the solr or RWI retrieval request - such as the content domain constraint) Changed Files: htroot/yacysearchtrailer.java, source/net/yacy/cora/federate/AbstractFederateSearchConnector.java, source/net/yacy/cora/sorting/ConcurrentScoreMap.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/util/ISO639.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/SearchEvent.java |
Thu Aug 31 07:37:24 CEST 2017 by luccioman | Use final results counts in progress bar detailed statistics. Using unfiltered detailed counts (local and remote entries found before doubles detection and before applying query modifiers) was confusing and inconsistent with the total count. It could let think more results are to come in the next pages, without understanding why they are not displayed. Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchlatestinfo.json, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Thu Aug 24 18:47:18 CEST 2017 by luccioman | Removed some unnecessary uses of java.lang.reflect api. This improves code browsing and readability, making search by references or call hierarchy IDE features more accurate. Changed Files: htroot/ConfigBasic.java, htroot/api/ymarks/import_ymark.java, source/net/yacy/contentcontrol/ContentControlFilterUpdateThread.java, source/net/yacy/contentcontrol/SMWListSyncThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/kelondro/workflow/OneTimeBusyThread.java, source/net/yacy/peers/OnePeerPingBusyThread.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/SearchEvent.java |
Mon Aug 21 09:38:20 CEST 2017 by luccioman | Improved parsing support for OOXML spreadsheets (.xlsx) As reported edycop in mantis 765 ( http://mantis.tokeek.de/view.php?id=765 ), parsing of xlsx files was quite incomplete. Now properly support "Shared String Table" entry in Office Open XML spreadsheets, an also detect embedded URLs. Integrating the Apache poi-ooxml library could be an option for finer OOXML formats support, but their SAX style parsing example ( http://poi.apache.org/spreadsheet/how-to.html#xssf_sax_api ) tends to show that a custom SAX handler is still efficient for lightweight and low memory footprint processing. Changed Files: source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/xml/GenericXMLContentHandler.java, source/net/yacy/document/parser/xml/OOXMLSharedStringsHandler.java, source/net/yacy/document/parser/xml/OOXMLSpreeadsheetHandler.java, test/java/net/yacy/document/ParserTest.java, test/java/net/yacy/document/parser/ooxmlParserTest.java, test/parsertest/umlaute_linux.ppsx, test/parsertest/umlaute_linux.xlsx |
Mon Aug 14 14:57:58 CEST 2017 by luccioman | Implemented partial stream parsing of tar archives. Also added JUnit tests for the tar parser and fixed unwanted use of the tar parser as a fallback on files included in a tar archive. Changed Files: source/net/yacy/document/parser/tarParser.java, test/java/net/yacy/document/parser/tarParserTest.java, test/parsertest/umlaute_dc_xml_iso.xml, test/parsertest/umlaute_dc_xml_utf8.xml, test/parsertest/umlaute_html_iso.html, test/parsertest/umlaute_html_utf8.html, test/parsertest/umlaute_html_xml_txt_gnu.tar, test/parsertest/umlaute_html_xml_txt_pax.tar, test/parsertest/umlaute_html_xml_txt_ustar.tar, test/parsertest/umlaute_html_xml_txt_v7.tar, test/parsertest/umlaute_linux.txt |
Fri Aug 11 20:50:36 CEST 2017 by luccioman | Fixed missing transitive dependency to commons-collections4-4.1 Dependency required by poi-3.16. Dependency was not provided in YaCy but already defined on previous poi versions. This only became problematic since upgrade from poi-3.15 to poi-3.16 (commit dedc6552d37b5e877258abddac9621f7fe75bf9b). Indeed in this new poi release, a poi component used in some YaCy parsers code paths now explicitely needs a class from the commons-collections4 library : org.apache.poi.hpsf.Section uses now org.apache.commons.collections4.bidimap.TreeBidiMap. Impacted YaCy parsers : xlsParser, pptParser, docParser. Issue detected by the folowing JUnit tests failing : ParserTest.testpptParsers(), ParserTest.testdocParsers(), xlsParserTest.testParse() Changed Files: .classpath, lib/commons-collections4-4.1.License, lib/commons-collections4-4.1.jar |
Sat Jul 08 09:04:03 CEST 2017 by luccioman | Started support of partial parsing on large streamed resources. Thus enable getpageinfo_p API to return something in a reasonable amount of time on resources over MegaBytes size range. Support added first with the generic XML parser, for other formats regular crawler limits apply as usual. Changed Files: htroot/api/getpageinfo_p.java, source/net/yacy/crawler/retrieval/StreamResponse.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Document.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/GenericXMLParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/xml/GenericXMLContentHandler.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/repository/LoaderDispatcher.java, test/java/net/yacy/document/parser/GenericXMLParserTest.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java |
Sat Jul 01 23:58:28 CEST 2017 by reger | upd to Jetty 9.4.6.v20170531 Modify loginservice to the changes in Jetty, partially based on pull request #101 https://github.com/yacy/yacy_search_server/pull/101 bu @automenta Changed Files: .classpath, build.xml, htroot/ConfigUser_p.java, lib/jetty-client-9.4.6.v20170531.jar, lib/jetty-continuation-9.4.6.v20170531.jar, lib/jetty-deploy-9.4.6.v20170531.jar, lib/jetty-http-9.4.6.v20170531.jar, lib/jetty-io-9.4.6.v20170531.jar, lib/jetty-jmx-9.4.6.v20170531.jar, lib/jetty-proxy-9.4.6.v20170531.jar, lib/jetty-security-9.4.6.v20170531.jar, lib/jetty-server-9.4.6.v20170531.jar, lib/jetty-servlet-9.4.6.v20170531.jar, lib/jetty-servlets-9.4.6.v20170531.jar, lib/jetty-util-9.4.6.v20170531.jar, lib/jetty-webapp-9.4.6.v20170531.jar, lib/jetty-xml-9.4.6.v20170531.jar, lib/jetty.License, pom.xml, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/http/MonitorHandler.java, source/net/yacy/http/YaCyLegacyCredential.java, source/net/yacy/http/YaCyLoginService.java |
Tue Jun 27 06:42:33 CEST 2017 by luccioman | Ensure lower case conversion consistency with any default locale. Especially for Turkish speaking users using "tr" as their system default locale : strings for technical stuff (URLs, tag names, constants...) must not be lower cased with the default locale, as 'I' doesn't becomes 'i' like in other locales such as "en", but becomes '?'. Changed Files: htroot/ConfigHeuristics_p.java, htroot/Crawler_p.java, htroot/api/blacklists/add_entry_p.java, htroot/api/blacklists/delete_entry_p.java, htroot/api/getpageinfo_p.java, htroot/api/ymarks/add_ymark.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/data/wiki/WikiCode.java, source/net/yacy/document/Document.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/http/AbstractRemoteHandler.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/util/Formatter.java, source/net/yacy/kelondro/util/ISO639.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphSchema.java, source/net/yacy/server/serverObjects.java, source/net/yacy/utils/translation/TranslatorXliff.java, source/net/yacy/yacy.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Mon Jun 26 16:30:21 CEST 2017 by luccioman | Added a generic XML parser, able to parse elements text and URLs. This parser adds support for any XML based format other than already supported XML vocabularies such XHTML, RSS/Atom feeds... It will eventually be used as a fallback if one of these specific parsers fail, before falling back to the existing genericParser which extracts not that much useful information except URL tokens. Changed Files: source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/GenericXMLParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/xml/GenericXMLContentHandler.java, source/net/yacy/kelondro/io/CharBuffer.java, test/java/net/yacy/document/parser/GenericXMLParserTest.java, test/parsertest/umlaute_dc_xml_iso.xml, test/parsertest/umlaute_dc_xml_utf8.xml |
Tue Jun 20 09:21:55 CEST 2017 by luccioman | Cleaned up memory usage page HTML - fixed validation errors - removed deprecated attributes - improved accessibility with richer table semantics (headers and caption elements) and language declaration Changed Files: htroot/PerformanceMemory_p.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/sk.lng, locales/uk.lng |
Wed Jun 14 09:13:50 CEST 2017 by luccioman | Limit the synchronization blocking time on some Cache operations. Using a Reentrant lock instead of the intrinsic synchronization lock permits limiting the blocking time to acquire a lock. Useful on a very busy Cache concurrently accessed by many threads : when the time to acquire a lock is too high, getting/storing content on the cache becomes inefficient, and it is then better to fall back to loading remote resources. Illustrated by the CacheTest stress test and some traces reported in mantis 751 ( http://mantis.tokeek.de/view.php?id=751 ) Changed Files: source/net/yacy/crawler/data/Cache.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/search/Switchboard.java, test/java/net/yacy/crawler/data/CacheTest.java |
Fri Jun 09 12:25:23 CEST 2017 by Michael Peter Christen | migrated Solr 5.5 -> Solr 6.6 and from Java 1.7 -> 1.8 Also: now Version 1.921 Changed Files: .classpath, .settings/org.eclipse.jdt.core.prefs, build.properties, build.xml, defaults/solr/schema.xml, defaults/solr/solrconfig.xml, htroot/yacysearchtrailer.java, lib/commons-math3-3.4.1.jar, lib/lucene-analyzers-common-6.6.0.jar, lib/lucene-analyzers-phonetic-6.6.0.jar, lib/lucene-backward-codecs-6.6.0.jar, lib/lucene-classification-6.6.0.jar, lib/lucene-codecs-6.6.0.jar, lib/lucene-core-6.6.0.jar, lib/lucene-facet-6.6.0.jar, lib/lucene-grouping-6.6.0.jar, lib/lucene-highlighter-6.6.0.jar, lib/lucene-join-6.6.0.jar, lib/lucene-memory-6.6.0.jar, lib/lucene-misc-6.6.0.jar, lib/lucene-queries-6.6.0.jar, lib/lucene-queryparser-6.6.0.jar, lib/lucene-spatial-6.6.0.jar, lib/lucene-suggest-6.6.0.jar, lib/metrics-core-3.2.2.jar, lib/solr-core-6.6.0.jar, lib/solr-dataimporthandler-6.6.0.jar, lib/solr-solrj-6.6.0.jar, lib/spatial4j-0.6.jar, lib/zookeeper-3.4.10.jar, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/cora/federate/solr/instance/ServerMirror.java, source/net/yacy/cora/federate/solr/instance/ServerShard.java, source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, test/java/net/yacy/document/DateDetectionTest.java |
Sat Jun 03 04:00:46 CEST 2017 by luccioman | Ensure file input streams proper closing in both success and failures Also add when possible a warning level log message on input stream closing error instead of failing silently. This could help understanding some IO exceptions such as "too many files open". Changed Files: source/net/yacy/document/parser/images/bmpParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/images/icoParser.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/kelondro/blob/Gap.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/util/XMLTables.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/AutoSearch.java, source/net/yacy/search/Switchboard.java, source/net/yacy/server/http/TemplateEngine.java, source/net/yacy/utils/PKCS12Tool.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/tarTools.java, source/net/yacy/utils/translation/TranslationManager.java, test/java/net/yacy/document/parser/htmlParserTest.java, test/java/net/yacy/document/parser/images/genericImageParserTest.java, test/java/net/yacy/document/parser/images/metadataImageParserTest.java, test/java/net/yacy/document/parser/pdfParserTest.java |
Fri Jun 02 12:14:29 CEST 2017 by luccioman | Ensure proper closing of file input streams. Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/geo/OpenGeoDBLocation.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/crawler/data/Snapshots.java, source/net/yacy/data/Translator.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/utils/CryptoLib.java, source/net/yacy/utils/PKCS12Tool.java, source/net/yacy/utils/cryptbig.java, source/net/yacy/utils/gzip.java, source/net/yacy/yacy.java, test/java/net/yacy/document/ParserTest.java, test/java/net/yacy/document/parser/xlsParserTest.java |
Fri Jun 02 01:00:21 CEST 2017 by reger | Introduce keyword query parameter This enables keyword navigator to filter on keywords. Added search page output and layout config for keywords, allowing e.g. in Intranet use to display the keywords. No styling or links applied to the keyword text (but is desirable possibly in combination with bootstrap-tagsinput for future/intranet). Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/index.html, htroot/yacysearchitem.html, htroot/yacysearchitem.java, source/net/yacy/search/navigator/StringNavigator.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Mon May 15 13:15:16 CEST 2017 by luccioman | Added user interface feedback on results feeding termination status. Added as an additional icon with title in the search progress bar, to inform about background search feeder threads terminated or still running. While giving a bit more information to users about the p2p search process, this can help choosing whether or not wait a little bit more time before going to the next page, in order to get results from various sources sorted as best as possible (see #91 for a discussion about sorting accuracy and network latency). Other related modifications included : - regular updates to statistics in the progress bar until the background feeders are completely terminated. - removed some uses of unsecure and discouraged JavaScript elements Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchlatestinfo.json, source/net/yacy/search/query/SearchEvent.java |
Thu May 11 18:02:33 CEST 2017 by luccioman | Improved previous merge "Show ranking in HTML UI". - added the new setting as configurable in the "Debug/Analysis" settings page. Debug/analysis is its main purpose for now as there is currently no nice and "understansable" ranking score info servlet (see forum discussion http://forum.yacy-websuche.de/viewtopic.php?f=8&t=5884 ) - render in the "Search Page Layout" page preview when enabled - added constants Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/SettingsAck_p.java, htroot/Settings_Debug.inc, htroot/Settings_p.java, htroot/yacysearchitem.html, source/net/yacy/search/SwitchboardConstants.java |
Fri Apr 14 14:32:44 CEST 2017 by luccioman | Extended Mediawiki dump import to remote URLs. When using a public HTTP URL in /IndexImportMediawiki_p.html, the remote file now is directly streamed and processed, allowing import of several GB dumps even with a low memory remote peer, and without need to manually download the dump file first. Changed Files: bin/importmediawiki.sh, htroot/IndexImportMediawiki_p.html, htroot/IndexImportMediawiki_p.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/index/DocumentIndex.java |
Thu Apr 06 21:18:01 CEST 2017 by reger | upd to Solr-5.5.4 Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-5.5.4.jar, lib/lucene-analyzers-phonetic-5.5.4.jar, lib/lucene-backward-codecs-5.5.4.jar, lib/lucene-classification-5.5.4.jar, lib/lucene-codecs-5.5.4.jar, lib/lucene-core-5.5.4.jar, lib/lucene-facet-5.5.4.jar, lib/lucene-grouping-5.5.4.jar, lib/lucene-highlighter-5.5.4.jar, lib/lucene-join-5.5.4.jar, lib/lucene-memory-5.5.4.jar, lib/lucene-misc-5.5.4.jar, lib/lucene-queries-5.5.4.jar, lib/lucene-queryparser-5.5.4.jar, lib/lucene-spatial-5.5.4.jar, lib/lucene-suggest-5.5.4.jar, lib/solr-core-5.5.4.jar, lib/solr-solrj-5.5.4.jar, pom.xml |
Tue Apr 04 00:59:26 CEST 2017 by reger | upd to pdfbox-2.0.5.jar and transient dependency xmpcore-5.1.3.jar required by metadata-extractor-2.10.1 (fix build.xml compiler warning) Changed Files: .classpath, build.xml, lib/fontbox-2.0.5.License, lib/fontbox-2.0.5.jar, lib/pdfbox-2.0.5.License, lib/pdfbox-2.0.5.jar, lib/xmpcore-5.1.3.jar, lib/xmpcore-5.1.3.license, pom.xml |
Mon Apr 03 11:34:49 CEST 2017 by luccioman | Set Config Portal as a private administration page. Consistently with its required action from submission credentials, and because external unauthenticated users do not need to access these settings. Changed Files: defaults/yacy.init, htroot/ConfigAppearance_p.html, htroot/ConfigPortal.java, htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/env/templates/header.template, htroot/env/templates/submenuPortalConfiguration.template, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, source/net/yacy/http/servlets/GSAsearchServlet.java |
Fri Mar 31 00:58:11 CEST 2017 by reger | Implement surrogate import from Warc archives (as first option handle warc = Web ARChive File Format. Warc files with extension .warc or compressed warc.gz can be placed in the DATA/surrogate/in and contained responses are imported to the index. The used library is stream based so we can easily extend it later to use and load warc's from the net. Changed Files: .classpath, build.xml, lib/jwat-archive-common-1.0.4.jar, lib/jwat-common-1.0.4.jar, lib/jwat-gzip-1.0.4.jar, lib/jwat-warc-1.0.4.jar, pom.xml, source/net/yacy/document/importer/WarcImporter.java, source/net/yacy/search/Switchboard.java |
Sun Mar 26 11:48:00 CEST 2017 by luccioman | Enforced access controls on some administrative actions. - ensure use of HTTP POST method : HTTP GET should only be used for information retrieval and not to perform server side effect operations (see HTTP standard https://tools.ietf.org/html/rfc7231#section-4.2.1) - a transaction token is now required for these administrative form submissions to ensure the request can not be included in an external site and performed silently/by mistake by the user browser Changed Files: bin/clearall.sh, bin/clearcache.sh, bin/clearindex.sh, bin/deleteurl.sh, bin/passwd.sh, bin/protectedPostApiCall.sh, htroot/ConfigAccounts_p.html, htroot/ConfigAccounts_p.java, htroot/ConfigProperties_p.html, htroot/ConfigProperties_p.java, htroot/ConfigUpdate_p.html, htroot/ConfigUpdate_p.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexDeletion_p.html, htroot/IndexDeletion_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/Performance_p.html, htroot/Steering.html, htroot/Steering.java, htroot/env/templates/header.template, htroot/terminal_p.html, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/data/BadTransactionException.java, source/net/yacy/data/TransactionManager.java, source/net/yacy/http/servlets/DisallowedMethodException.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/yacy.java, stopYACY.sh |
Tue Mar 21 17:15:01 CET 2017 by luccioman | Updated shell scripts to be compatible with HTTP Digest authentication Because curl and wget do not let use a hashed password as parameter, YaCy shell scripts which require authentication are now interactive by default when HTTP Digest is the only available authentication method. Batch mode can still be available trough the use of an environment variable : YACY_ADMIN_PASSWORD. Other improvements : - added backward compatibility for Basic Authentication - fixed curl/wget presence detection - do not return with exit code 0 when an API call failed, and print an error message when the case occurs - documented available authentication options for API calls Changed Files: bin/apicall.sh, bin/apicat.sh, bin/down.sh, bin/passwd.sh, bin/search1.sh, stopYACY.sh |
Sun Mar 19 02:30:08 CET 2017 by reger | Introduce the option to configure a shutdown port. A port value of -1 will disable this option. If set to a value greater 0, YaCy listens on this of on the local loopback address (127.0.0.1) for a shutdown or restart signal. E.g. connect to http://localhost:8005/shutdown will stop the YaCy server. http://localhost:8005/restart will restart it. This option allows to stop YaCy locally independant from the web web frontend (which might be configured for password protected remote access). Changed Files: defaults/yacy.init, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_ServerAccess.inc, htroot/Settings_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/server/serverSwitch.java |
Sat Mar 18 20:02:26 CET 2017 by reger | add switchboardconstants for server ports config keys Changed Files: htroot/ConfigBasic.java, htroot/QuickCrawlLink_p.java, htroot/SettingsAck_p.java, htroot/api/snapshot.java, source/net/yacy/gui/Tray.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/migration.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Seed.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/utils/upnp/UPnP.java, source/net/yacy/yacy.java |
Tue Feb 28 18:11:54 CET 2017 by luccioman | Privacy enhancement : added settings to control referrer policy. HTTP "Referer" header sent by the browser when using YaCy can now be controlled either with the referrer meta tag as a global policy, or only for search result links by adding the attribute rel="noreferrer". To improve privacy with the less possible regressions, the default is set as meta tag with value "origin-when-cross-origin" : internal YaCy links behavior is not affected, but when visiting external websites referrer url is not empty but stripped from query parameters and path. Older browsers, Safari, MS IE and Edge do not support the referrer meta tag, so the standard but less flexible noreferrer link type can also be enabled as an alternative. User-friendly settings page to be implemented. Changed Files: defaults/yacy.init, htroot/env/templates/metas.template, htroot/yacysearchitem.html, htroot/yacysearchitem.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Feb 20 10:48:07 CET 2017 by luccioman | Refactored and enforced Solr mandatory fields for proper operation - Added a new method to check activation of mandatory fields on Collection Configuration commit, consistently with checks previously performed in Switchboard startup and with mandatory fields in the default schema. - Reorganized default schema and CollectionConfiguration enumeration : moved no more mandatory fields in a specific section, and moved fields enabled at startup to the mandatory section. - Marked mandatory fields as required and with stronger font in the IndexSchema_p.html page Changed Files: defaults/solr.collection.schema, htroot/IndexSchema_p.html, htroot/IndexSchema_p.java, source/net/yacy/cora/federate/solr/SchemaDeclaration.java, source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java, source/net/yacy/search/schema/WebgraphSchema.java |
Mon Feb 13 19:11:17 CET 2017 by luccioman | Added support for HTML OpenSearch results. Many OpenSearch systems do not provide results as standard RSS/Atom feeds but only as HTML. This modification add some support for custom OpenSearch HTML results through the use of mapping files (as already done for federated Solr search) relying on CSS-like selectors to retrieve information from HTML content. An example mapping file is provided to map results from the www.npmjs.com OpenSearch URL. Changed Files: defaults/federatecfg/npmjs.html.map.properties, defaults/heuristicopensearch.conf, source/net/yacy/cora/federate/AbstractFederateSearchConnector.java, source/net/yacy/cora/federate/FederateSearchManager.java, source/net/yacy/cora/federate/opensearch/OpenSearchConnector.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/http/HTTPClient.java |
Sat Feb 11 19:53:27 CET 2017 by reger | upd to Jetty-9.2.21.v20170120 Changed Files: .classpath, build.xml, lib/jetty-client-9.2.21.v20170120.jar, lib/jetty-continuation-9.2.21.v20170120.jar, lib/jetty-deploy-9.2.21.v20170120.jar, lib/jetty-http-9.2.21.v20170120.jar, lib/jetty-io-9.2.21.v20170120.jar, lib/jetty-jmx-9.2.21.v20170120.jar, lib/jetty-proxy-9.2.21.v20170120.jar, lib/jetty-security-9.2.21.v20170120.jar, lib/jetty-server-9.2.21.v20170120.jar, lib/jetty-servlet-9.2.21.v20170120.jar, lib/jetty-servlets-9.2.21.v20170120.jar, lib/jetty-util-9.2.21.v20170120.jar, lib/jetty-webapp-9.2.21.v20170120.jar, lib/jetty-xml-9.2.21.v20170120.jar, pom.xml |
Thu Feb 09 11:05:06 CET 2017 by luccioman | Added a new Debug/Analysis advanced settings subsection. As discussed in PR #93 with @JeremyRand and @reger24 this new advanced settings page includes: - a new setting to control remote Solr responses encoding - some existing debug settings which could not be set through the admin user interface Changed Files: defaults/yacy.init, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_Debug.inc, htroot/Settings_p.html, htroot/Settings_p.java, source/net/yacy/cora/federate/SolrFederateSearchConnector.java, source/net/yacy/cora/federate/solr/instance/InstanceMirror.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/AutoSearch.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java |
Fri Jan 27 15:47:15 CET 2017 by luccioman | Added user-friendly controls over disk usage configuration settings. As mentioned in issue #103, control settings over YaCy disk usage already existed but lacked a user-friendly way to set them. I added it to the Performance_p.html administration page with a little refactoring on the "Resource Observer" fieldset for improved accessibility and HTML standards respect. Also added the possibility to enable/disable the autoregulation fonction from this page. Changed Files: htroot/PerformanceQueues_p.java, htroot/Performance_p.html, htroot/env/base.css, locales/cn.lng, locales/de.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, source/net/yacy/search/ResourceObserver.java, source/net/yacy/search/SwitchboardConstants.java |
Sun Jan 22 23:58:46 CET 2017 by reger | Group all proxy settings on System Administration by adding settings of UrlProxyAccss page (moved from deleted AugmentedBrowsing_p), adjust submenu (remove Augmented Browsing) and translation files. Changed Files: htroot/ConfigSearchPage_p.html, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_UrlProxyAccess.inc, htroot/Settings_p.html, htroot/Settings_p.java, htroot/Status_p.inc, htroot/env/templates/submenuSemantic.template, locales/de.lng, locales/fr.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, source/net/yacy/http/servlets/UrlProxyServlet.java, source/net/yacy/http/servlets/YaCyProxyServlet.java |
Sat Jan 21 00:26:04 CET 2017 by reger | upd to solr-5.5.3 minor bugfix version Changed Files: .classpath, build.xml, lib/lucene-analyzers-common-5.5.3.jar, lib/lucene-analyzers-phonetic-5.5.3.jar, lib/lucene-backward-codecs-5.5.3.jar, lib/lucene-classification-5.5.3.jar, lib/lucene-codecs-5.5.3.jar, lib/lucene-core-5.5.3.jar, lib/lucene-facet-5.5.3.jar, lib/lucene-grouping-5.5.3.jar, lib/lucene-highlighter-5.5.3.jar, lib/lucene-join-5.5.3.jar, lib/lucene-memory-5.5.3.jar, lib/lucene-misc-5.5.3.jar, lib/lucene-queries-5.5.3.jar, lib/lucene-queryparser-5.5.3.jar, lib/lucene-spatial-5.5.3.jar, lib/lucene-suggest-5.5.3.jar, lib/solr-core-5.5.3.jar, lib/solr-solrj-5.5.3.jar, pom.xml |
Mon Jan 09 16:44:47 CET 2017 by luccioman | Cleaned up some Javadoc warnings. Changed Files: source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/data/list/ListAccumulator.java, source/net/yacy/data/list/XMLBlacklistImporter.java, source/net/yacy/data/ymark/YMarkUtil.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LargeNumberCache.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/Parser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/importer/Importer.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/apkParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/Evaluation.java, source/net/yacy/document/parser/html/ImageEntry.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/gui/framework/Switchboard.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/navigator/LanguageNavigator.java, source/net/yacy/search/navigator/Navigator.java, source/net/yacy/search/navigator/RestrictedStringNavigator.java, source/net/yacy/search/navigator/YearNavigator.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/snippet/TextSnippet.java |
Wed Jan 04 17:09:37 CET 2017 by luccioman | Upgraded jgit build library to version 4.5.0 This is the latest Java 7 compatible jgit release. Properly support GitHub tags marked as "Pre-release". With the previous venerable jgit version 1.1.0, a YaCy repository clone having such a tag made GitRevTask and GitRevMavenTask crash. Changed Files: build.xml, libbuild/GitRevMavenTask/pom.xml, libbuild/GitRevMavenTask/src/GitRevMavenTask.java, libbuild/GitRevTask/GitRevTask.java, libbuild/JavaEWAH-0.7.9.License, libbuild/JavaEWAH-0.7.9.jar, libbuild/httpclient-4.3.6.License, libbuild/httpclient-4.3.6.jar, libbuild/jsch-0.1.53.License, libbuild/jsch-0.1.53.jar, libbuild/org.eclipse.jgit-4.5.0.201609210915-r.License, libbuild/org.eclipse.jgit-4.5.0.201609210915-r.jar, libbuild/slf4j-api-1.7.2.License, libbuild/slf4j-api-1.7.2.jar, pom.xml |
Commit | Description |
---|---|
Wed Dec 09 02:22:47 CET 2020 by Michael Peter Christen | javadoc fix Changed Files: source/net/yacy/search/Switchboard.java |
Wed Dec 09 02:22:30 CET 2020 by Michael Peter Christen | enhanced ability to debug the code Changed Files: source/net/yacy/http/YaCyLegacyCredential.java |
Thu Dec 03 00:40:51 CET 2020 by Michael Peter Christen | missing fix for latest commit Changed Files: source/net/yacy/http/InetPathAccessHandler.java |
Thu Nov 26 20:52:46 CET 2020 by Radek Podgorny | typo fix Changed Files: docker/Readme.md |
Sun Nov 22 23:12:04 CET 2020 by Michael Peter Christen | some debug lines Changed Files: source/net/yacy/kelondro/util/MemoryStrategy.java |
Wed Jul 22 22:15:00 CEST 2020 by sgaebel | fixes deleting during recrawl Changed Files: source/net/yacy/crawler/RecrawlBusyThread.java |
Mon Jan 20 21:01:45 CET 2020 by sgaebel | fix case esac in start.sh Changed Files: startYACY.sh |
Mon Jan 20 14:11:03 CET 2020 by parnikkapore | Typo fix Changed Files: defaults/yacy.network.readme |
Thu Jan 16 19:48:52 CET 2020 by Michael Peter Christen | build path fix Changed Files: .classpath |
Wed Nov 06 17:28:11 CET 2019 by Michael Peter Christen | fix for bad json Changed Files: htroot/Network.java, htroot/Network.json |
Sun Oct 20 20:20:50 CEST 2019 by Michael Christen | fixed links Changed Files: htroot/Steering.html, htroot/env/templates/header.template, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, pom.xml, source/net/yacy/search/index/Segment.java |
Mon Oct 14 01:53:09 CEST 2019 by Michael Christen | fixed css profile name Changed Files: defaults/yacy.init |
Sun Oct 13 11:12:32 CEST 2019 by Michael Peter Christen | fix for failing parser test: new forum link Changed Files: test/parsertest/umlaute_linux.xlsx |
Sat Jun 15 11:43:27 CEST 2019 by Michael Christen | fixed many links to old forum, now https://searchlab.eu Changed Files: README.md, htroot/Status.html, htroot/Steering.html, htroot/env/templates/header.template, htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, htroot/yacy/ui/yacyui-admin.html, htroot/yacy/ui/yacyui-welcome.html, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng, startYACY.sh, startYACY_debug.bat, test/java/net/yacy/document/parser/GenericXMLParserTest.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java, test/java/net/yacy/document/parser/ooxmlParserTest.java |
Wed Feb 13 22:18:59 CET 2019 by luccioman | Minor technical fixes on latest Chinese translation update Changed Files: locales/zh.lng |
Sat Feb 09 15:43:15 CET 2019 by luccioman | Fixed Alpine flavour Docker image build on already existing /opt folder Changed Files: docker/Dockerfile.alpine |
Mon Dec 17 14:12:21 CET 2018 by luccioman | Fixed a case of NullPointerException on disconnected RWI data structure Changed Files: htroot/api/termlist_p.java |
Mon Dec 10 18:23:46 CET 2018 by luccioman | Updated the down.sh script, fixing the same kind of issue as in PR #260 Changed Files: bin/down.sh |
Mon Dec 10 17:51:15 CET 2018 by luccioman | Fixed broken searchall shell scripts Changed Files: bin/searchall.sh, bin/searchall1.sh |
Sat Dec 01 10:55:15 CET 2018 by luccioman | Removed shell temporary debug echo message Changed Files: bin/passwd.sh |
Sun Nov 18 10:03:41 CET 2018 by luccioman | Fixed detection of absolute data folder path on MS Windows Changed Files: source/net/yacy/yacy.java |
Thu Oct 11 18:28:08 CEST 2018 by luccioman | Added documentation hint about JVM option useful to debug JVM crashes Changed Files: build.xml |
Tue Oct 02 07:32:36 CEST 2018 by luccioman | Additional modifications for typo fix in Bookmarks.html from PR #240 Changed Files: locales/de.lng, locales/ja.lng, locales/master.lng.xlf, locales/zh.lng |
Tue Aug 28 15:00:18 CEST 2018 by luccioman | Fixed portal search widget results favicon url Changed Files: htroot/portalsearch/yacy-portalsearch.js |
Mon Aug 13 14:36:22 CEST 2018 by luccioman | Fixed rendering of crawl queues page for URLs with raw IPV6 addresses Changed Files: htroot/IndexCreateQueues_p.java |
Mon Aug 13 14:35:26 CEST 2018 by luccioman | Fixed NullPointerException case on malformed crawl queue folder name Changed Files: source/net/yacy/cora/protocol/Domains.java, source/net/yacy/crawler/HostBalancer.java, source/net/yacy/crawler/data/NoticedURL.java |
Mon Aug 06 12:04:44 CEST 2018 by luccioman | Fixed a NullPointerException case, reported in issue #209 Changed Files: htroot/ConfigHeuristics_p.java |
Thu Jul 26 08:27:21 CEST 2018 by luccioman | Fixed few technical mistakes in updated Chinese translation from PR #188 Changed Files: locales/zh.lng |
Tue Jul 17 07:08:30 CEST 2018 by luccioman | Fixed minor technical issues on Chinese updated translations Changed Files: locales/zh.lng |
Sun Jul 08 08:15:26 CEST 2018 by luccioman | Fixed small unnecessary lines of code Changed Files: source/net/yacy/gui/framework/Switchboard.java |
Sun Jul 08 08:11:01 CEST 2018 by luccioman | Fixed useless comparison between int parameter and Long.MAX_VALUE Changed Files: source/net/yacy/peers/Dispatcher.java |
Fri Jun 29 13:01:05 CEST 2018 by luccioman | Fixed JavaScript sorting of tables with cells containing an input field Changed Files: htroot/js/sorttable.js |
Wed Jun 20 07:51:59 CEST 2018 by luccioman | Fixed potential infinite loop case (does not occur in current code base) Changed Files: source/net/yacy/cora/protocol/RequestHeader.java |
Tue Jun 19 12:50:28 CEST 2018 by luccioman | Fixed rendering of solr mustNoMatch value on CrawlProfileEditor_p.xml Changed Files: source/net/yacy/crawler/data/CrawlProfile.java |
Tue Jun 19 11:58:47 CEST 2018 by luccioman | Fixed default crawl profile solr mustnotmatch query from previous commit Changed Files: source/net/yacy/crawler/data/CrawlProfile.java |
Tue Jun 19 07:07:18 CEST 2018 by luccioman | Fixed JUnit test after removal of unused Transformer Changed Files: test/java/net/yacy/document/parser/html/ContentScraperTest.java |
Tue Jun 12 14:29:18 CEST 2018 by luccioman | Additional minor fix in Italian translation Changed Files: locales/it.lng |
Tue Jun 12 14:18:29 CEST 2018 by luccioman | Fixed various minor mistakes in italian translation Changed Files: locales/it.lng |
Sat Jun 02 14:54:05 CEST 2018 by luccioman | Fixed unresolve-pattern case on old html title Changed Files: htroot/Connections_p.html |
Fri May 18 11:30:14 CEST 2018 by luccioman | Fixed NullPointerException when omitHeader=true on external Solr server Changed Files: source/net/yacy/cora/federate/solr/instance/ResponseAccumulator.java |
Fri May 18 11:28:30 CEST 2018 by luccioman | Fixed Solr response header duplication when requesting external Solr Changed Files: source/net/yacy/http/servlets/SolrSelectServlet.java |
Mon May 14 18:26:25 CEST 2018 by luccioman | Fixed largest snippet get : don't reject ones starting with a space char Changed Files: source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java |
Mon May 14 15:21:21 CEST 2018 by luccioman | Fixed snippets statistics displayed "provided by Solr" count Changed Files: htroot/ConfigPortal_p.html |
Tue May 01 09:39:41 CEST 2018 by luccioman | Fixed text snippet max init time statistic rendering Changed Files: source/net/yacy/search/snippet/TextSnippetStatistics.java |
Tue Mar 27 08:25:40 CEST 2018 by luccioman | Fixed initial ViewFile mode and suggestions links from previous commit Changed Files: htroot/ViewFile.html, htroot/ViewFile.java |
Wed Feb 28 12:23:52 CET 2018 by luccioman | Small fix on svg parser error message Changed Files: source/net/yacy/document/parser/images/svgParser.java |
Wed Feb 28 07:31:32 CET 2018 by luccioman | Fixed NPE case when on audio resource parsed with null tag Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Sat Feb 10 11:56:28 CET 2018 by luccioman | Fixed issue #158 : completed div CSS class ignore in crawl Changed Files: htroot/CrawlStartExpert.html, source/net/yacy/document/parser/html/AbstractScraper.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/Scraper.java, source/net/yacy/document/parser/html/TransformerWriter.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Thu Feb 08 14:31:26 CET 2018 by luccioman | Fixed loss of search modifiers on bookmark, recommand or delete result Changed Files: htroot/yacysearchitem.java |
Tue Feb 06 17:17:13 CET 2018 by luccioman | Fixed loss of other modifiers on keywords/tags search navigation links Changed Files: source/net/yacy/search/query/QueryParams.java, test/java/net/yacy/search/query/QueryParamsTest.java |
Sat Jan 13 10:45:00 CET 2018 by luccioman | Use a constant for crawler reject reason prefix with specific processing Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/search/Switchboard.java |
Wed Jan 10 18:38:42 CET 2018 by luccioman | Fixed internal tables exact value match iterator Changed Files: source/net/yacy/kelondro/blob/Tables.java |
Fri Dec 01 09:48:42 CET 2017 by luccioman | Fixed URL parsing with fragment and empty path Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Thu Nov 30 20:21:45 CET 2017 by luccioman | Fixed url mask filter generated when protocol modifier is not null Changed Files: source/net/yacy/search/query/QueryParams.java |
Tue Oct 31 07:52:30 CET 2017 by luccioman | Fixed spelling Changed Files: htroot/index.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Tue Oct 24 09:30:21 CEST 2017 by luccioman | Fixed blacklist returned location URL on empty parameters Changed Files: source/net/yacy/repository/BlacklistHelper.java |
Wed Oct 18 08:31:18 CEST 2017 by luccioman | Fixed NullPointerException cases on snapshot images parsing. Changed Files: htroot/api/snapshot.java, source/net/yacy/cora/util/Html2Image.java |
Mon Oct 16 19:47:18 CEST 2017 by luccioman | Fixed a NullPointerException case on images encoding errors. Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Thu Oct 05 14:44:33 CEST 2017 by luccioman | Fixed Travis configuration for Debian package building task Changed Files: .travis.yml, build.xml |
Thu Oct 05 14:44:33 CEST 2017 by luccioman | Fixed Travis configuration for Debian package building task Changed Files: .travis.yml |
Thu Oct 05 14:26:55 CEST 2017 by luccioman | Fixed YaCy Debian package path in Travis configuration Changed Files: .travis.yml |
Tue Aug 29 07:32:33 CEST 2017 by luccioman | Fixed Unresolved_Pattern occurence on results favicon HTML id. Changed Files: htroot/yacysearchitem.java |
Sun Jul 16 14:39:53 CEST 2017 by luccioman | Distinguish response parsing failures from unexpected exceptions. Changed Files: source/net/yacy/crawler/retrieval/Response.java |
Tue Jul 11 09:00:27 CEST 2017 by luccioman | Fixed read/copy on input streams reading sometimes less than expected. Changed Files: source/net/yacy/kelondro/util/FileUtils.java, test/java/net/yacy/kelondro/util/FileUtilsTest.java |
Sat Jul 08 22:46:15 CEST 2017 by reger | Fix unresolved pattern in api/share.html by init some display var's Changed Files: htroot/api/share.java |
Fri Jun 30 01:06:17 CEST 2017 by luccioman | Do not wrap unnecessarily loader IOExceptions in IOExceptions Changed Files: source/net/yacy/repository/LoaderDispatcher.java |
Thu Jun 08 07:19:16 CEST 2017 by luccioman | Properly close file output streams even on exceptions scenarios. Changed Files: htroot/ConfigLanguage_p.java, source/net/yacy/cora/federate/solr/instance/EmbeddedInstance.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/storage/ZIPWriter.java, source/net/yacy/crawler/data/Transactions.java, source/net/yacy/data/Translator.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/parser/apkParser.java, source/net/yacy/document/parser/bzipParser.java, source/net/yacy/document/parser/gzipParser.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/kelondro/blob/Gap.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/index/BinSearch.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/util/XMLTables.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/AutoSearch.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/server/serverSwitch.java, source/net/yacy/utils/gzip.java, source/net/yacy/utils/tarTools.java, source/net/yacy/utils/translation/TranslatorXliff.java, source/net/yacy/visualization/AnimationGIF.java, source/net/yacy/visualization/AnimationPlotter.java, source/net/yacy/visualization/ChartPlotter.java, source/net/yacy/visualization/RasterPlotter.java |
Tue May 30 12:32:14 CEST 2017 by luccioman | Fix unescape of URLs having some '%' chars but not percent-encoded Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Tue May 30 08:48:20 CEST 2017 by luccioman | Fixed scraper NullPointerException cases on malformed URLs. Changed Files: source/net/yacy/document/parser/html/ContentScraper.java |
Thu May 18 00:28:12 CEST 2017 by Michael Peter Christen | enhanced debugging Changed Files: source/net/yacy/search/schema/CollectionSchema.java |
Tue May 09 12:15:41 CEST 2017 by luccioman | Fixed Debian install message misspelling. Changed Files: debian/yacy.templates |
Thu May 04 08:45:30 CEST 2017 by luccioman | Fixed the previously added link to scheduled dump operations. Changed Files: htroot/IndexImportMediawiki_p.html |
Mon May 01 11:44:26 CEST 2017 by Michael Peter Christen | copied fix from yacy_grid_parser for wrong array type Changed Files: source/net/yacy/document/parser/html/ContentScraper.java |
Mon Apr 24 13:27:07 CEST 2017 by luccioman | Fixed "Unchecked conversion" compilation warnings. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/util/JSONArray.java, source/net/yacy/cora/util/JSONObject.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/search/navigator/FileTypeNavigator.java, source/net/yacy/search/navigator/HostNavigator.java, source/net/yacy/search/navigator/StringNavigator.java, source/net/yacy/search/navigator/TokenizedStringNavigator.java, source/net/yacy/search/navigator/YearNavigator.java |
Fri Apr 14 21:14:26 CEST 2017 by reger | fix unresolved_pattern on missing post parameter api/message.html Changed Files: htroot/yacy/message.java |
Thu Mar 30 15:41:14 CEST 2017 by luccioman | Fixed NPE case and API URL link on Solr HTML output for webgraph core. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Tue Mar 07 12:27:27 CET 2017 by luccioman | Fixed settingsAck_p.html back link for case where referrer is stripped. Changed Files: htroot/SettingsAck_p.java |
Fri Mar 03 13:46:44 CET 2017 by luccioman | Fixed unresolved pattern case on /yacysearchlatestinfo.json api Changed Files: htroot/yacysearchlatestinfo.java |
Thu Feb 16 02:36:24 CET 2017 by reger | fix NPE in HTMLResponseWriter on missing document title Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Thu Feb 09 10:59:41 CET 2017 by luccioman | Fixed NPE case occurring when local solr index is disabled in search. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Jan 24 11:49:15 CET 2017 by luccioman | Index Browser : fixed display of "Count colors" for authorized users. Changed Files: htroot/HostBrowser.java |
Mon Jan 23 14:54:37 CET 2017 by luccioman | Fixed "-UNRESOLVED_PATTERN-" admin parameter in "load & index" links. Changed Files: htroot/HostBrowser.java |
Sat Jan 21 00:35:05 CET 2017 by reger | fix the missing solr-5.5.2.jar delete from prev. commit Changed Files: |
Mon Jan 09 17:59:01 CET 2017 by luccioman | Fixed 2 failing JUNit tests. Changed Files: test/java/net/yacy/document/DateDetectionTest.java, test/java/net/yacy/utils/translation/TranslatorXliffTest.java |
Mon Jan 09 09:57:53 CET 2017 by luccioman | Fixed some JavaDocs broken links. Changed Files: source/net/yacy/cora/bayes/Classifier.java, source/net/yacy/data/list/ListAccumulator.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Jan 09 09:54:14 CET 2017 by luccioman | Fixed maven assembly base directory to match last main YaCy binaries. Changed Files: assembly.xml |
Commit | Description |
---|---|
Mon Dec 14 13:20:28 CET 2020 by Michael Peter Christen | panic release for #googledown Changed Files: build.properties |
Mon Dec 14 02:08:49 CET 2020 by Michael Peter Christen | beautified default peer names Changed Files: defaults/yacy.init, source/net/yacy/peers/Seed.java, source/net/yacy/search/Switchboard.java |
Mon Dec 14 01:26:26 CET 2020 by Michael Peter Christen | automatically refresh css cache when switching skin and setting of default skin to current skin in selector Changed Files: htroot/ConfigAppearance_p.html, htroot/ConfigAppearance_p.java |
Sat Dec 12 14:17:45 CET 2020 by Michael Peter Christen | design update more space Changed Files: htroot/Status_p.inc, htroot/env/base.css, htroot/env/bootstrap-base.css, skins/pdblue.css |
Sat Dec 12 03:02:40 CET 2020 by Michael Peter Christen | using a lock symbol which looks also good in dark designs Changed Files: htroot/env/base.css, htroot/env/grafics/lockclose.png, htroot/env/grafics/lockopen.png |
Thu Dec 10 11:19:25 CET 2020 by Michael Peter Christen | fixed warc importer The importer tried to import a gziped files as plain warc. It will now check the file extension and use a unzip automatically on-the-fly. Changed Files: htroot/IndexImportWarc_p.java, source/net/yacy/document/importer/WarcImporter.java |
Wed Dec 09 02:42:05 CET 2020 by Michael Peter Christen | added a hint to the default settings how to set a default password Changed Files: defaults/yacy.init |
Sun Dec 06 01:49:34 CET 2020 by Michael Peter Christen | prevent creation of auth schema factories every time a servlet is called Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Thu Dec 03 12:18:07 CET 2020 by Michael Peter Christen | enhanced the (already fast!) png exporter Changed Files: source/net/yacy/visualization/RasterPlotter.java |
Wed Dec 02 18:39:45 CET 2020 by Michael Peter Christen | reduced complexity - removed concurrency in sort Changed Files: source/net/yacy/cora/sorting/Array.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/yacy.java |
Wed Dec 02 18:05:38 CET 2020 by Michael Peter Christen | adding message to UI to make port change transparent Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, source/net/yacy/http/Jetty9HttpServerImpl.java |
Wed Dec 02 00:13:00 CET 2020 by Michael Peter Christen | added a gc to cleanup process (once every 10 minutes) Changed Files: source/net/yacy/search/Switchboard.java |
Tue Dec 01 03:00:24 CET 2020 by Michael Peter Christen | creating a threaddump during every cleanup process to be able to find out what a peer did (not) last time before a crash Changed Files: htroot/Threaddump_p.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/search/Switchboard.java |
Tue Dec 01 02:18:08 CET 2020 by Michael Peter Christen | do better documentation on how to set a default password Changed Files: docker/Dockerfile, source/net/yacy/cora/order/Digest.java |
Sun Nov 29 14:24:33 CET 2020 by Michael Peter Christen | try to fix maven build error Changed Files: source/net/yacy/cora/protocol/http/auth/YaCyDigestSchemeFactory.java |
Sun Nov 29 14:15:34 CET 2020 by Michael Peter Christen | trying to fight the UI unavaiability this path addresses a possible issue with too many open connections to remote peers Changed Files: source/net/yacy/http/Jetty9HttpServerImpl.java |
Thu Nov 26 23:28:33 CET 2020 by Michael Christen | Merge pull request #390 from rpodgorny/patch-1 typo fix Changed Files: docker/Readme.md |
Mon Nov 23 00:03:18 CET 2020 by Michael Peter Christen | removed all remaining deprecation warnings Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/auth/YaCyDigestScheme.java, source/net/yacy/cora/protocol/http/auth/YaCyDigestSchemeFactory.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Sun Nov 22 23:27:58 CET 2020 by Michael Peter Christen | removed one deprecation warning for jetty library initializing ssl server port Changed Files: source/net/yacy/http/Jetty9HttpServerImpl.java |
Sun Nov 22 23:02:49 CET 2020 by Michael Peter Christen | showing MB instead of KB in PerformanceMemory Changed Files: htroot/PerformanceMemory_p.html, htroot/PerformanceMemory_p.java |
Mon Aug 03 21:06:46 CEST 2020 by sgaebel | removes unused 'unused' tokens Changed Files: htroot/IndexSchema_p.java, htroot/proxymsg/urlproxyheader.java, htroot/sharedBlacklist_p.java, htroot/yacysearchlatestinfo.java, source/net/yacy/search/schema/HyperlinkGraph.java |
Sun Aug 02 20:14:56 CEST 2020 by sgaebel | removes some warning and unused objects Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/utils/translation/TranslatorXliff.java, test/java/net/yacy/cora/document/feed/RSSFeedTest.java, test/java/net/yacy/cora/document/id/DigestURLHashPerfTest.java, test/java/net/yacy/search/ranking/ReferenceOrderTest.java |
Wed Jul 29 00:11:41 CEST 2020 by sgaebel | removes some deprecation-warnings Changed Files: source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/search/Switchboard.java, test/java/net/yacy/http/servlets/YaCyDefaultServletTest.java |
Sun Jul 12 16:57:46 CEST 2020 by Michael Peter Christen | added debug code to parser test to investigate why this fails in travis build Changed Files: test/java/net/yacy/document/parser/ooxmlParserTest.java |
Thu Jul 09 19:32:16 CEST 2020 by sgaebel | adds deleting during recrawl Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Fri Apr 24 11:42:06 CEST 2020 by Michael Peter Christen | modified org.json.* library to fit into the YaCy environment as drop-in replacement. Also made some fixes and enhancements to the library. Changed Files: source/org/json/JSON.java, source/org/json/JSONArray.java, source/org/json/JSONException.java, source/org/json/JSONObject.java, source/org/json/JSONStringer.java, source/org/json/JSONTokener.java |
Fri Apr 24 10:28:43 CEST 2020 by Michael Peter Christen | added org.json.* library from https://android.googlesource.com/platform/libcore/+/refs/heads/master/json/src/main/java/org/json as a preparation step for https://github.com/yacy/yacy_search_server/issues/347 Changed Files: source/org/json/JSON.java, source/org/json/JSONArray.java, source/org/json/JSONException.java, source/org/json/JSONObject.java, source/org/json/JSONStringer.java, source/org/json/JSONTokener.java |
Wed Apr 22 13:14:26 CEST 2020 by Michael Peter Christen | removed JSON-evil license line These classes had been my own creative work. Just the copyright line had been appeared possibly due to a bad copy-paste activity, unaware that the line is a non-free addition. Changed Files: htroot/api/trail_p.java, htroot/cytag.java |
Tue Mar 10 17:03:10 CET 2020 by Michael Christen | Merge pull request #338 from LoveIsGrief/docker-rename Replace all mentions of luccioman in docker Readme Changed Files: docker/Readme.md |
Sun Mar 08 18:59:00 CET 2020 by LoveIsGrief | Replace all mentions of luccioman in docker Readme It's not the official image Changed Files: docker/Readme.md |
Tue Feb 18 14:40:50 CET 2020 by Michael Christen | xss protection Changed Files: htroot/yacysearch.java, htroot/yacysearchtrailer.java |
Tue Feb 11 13:47:40 CET 2020 by Michael Christen | Merge pull request #333 from comradekingu/patch-2 YaCy packages README reworked Changed Files: yacy-packages.readme |
Tue Feb 11 13:47:04 CET 2020 by Michael Christen | Merge pull request #334 from danielthegray/fix_readme_formatting Fix formatting errors in the README file Changed Files: README.md |
Mon Feb 10 23:49:55 CET 2020 by Daniel Gray | Fix formatting errors in the README file Changed Files: README.md |
Fri Feb 07 17:46:04 CET 2020 by Allan Nordhøy | YaCy packages README reworked Changed Files: yacy-packages.readme |
Fri Feb 07 09:19:21 CET 2020 by Michael Christen | Merge pull request #327 from parnikkapore/pr_typos Typo fix in yacy.network.readme Changed Files: defaults/yacy.network.readme |
Fri Feb 07 09:18:54 CET 2020 by Michael Christen | Merge pull request #332 from comradekingu/patch-1 README reworked Changed Files: README.md |
Thu Feb 06 14:49:31 CET 2020 by Allan Nordhøy | README reworked Changed Files: README.md |
Sun Jan 19 04:42:03 CET 2020 by reger | upd windows installer script to Java 8 update 211 Changed Files: build.nsi |
Fri Jan 17 23:19:37 CET 2020 by Michael Christen | Merge pull request #325 from quietok/alternate-bsd-start Start script run on FreeBSD and likely other BSDs Changed Files: startYACY.sh |
Fri Jan 17 19:36:04 CET 2020 by Staff Silence | Start script run on FreeBSD and likely other BSDs Fix to make start script run on FreeBSD that i used, likely applies to dragonfly, trueos and possibly netbsd- but unable to test. Changed Files: startYACY.sh |
Thu Jan 16 21:09:42 CET 2020 by Michael Peter Christen | removed doubes from getpageinfo Changed Files: htroot/api/getpageinfo_p.java |
Thu Jan 16 20:59:02 CET 2020 by Michael Peter Christen | enhanced crawl start url check experience urls are now urlencoded and a check is also performed in case that an url is copied into the url field using copy-paste Changed Files: htroot/CrawlStartExpert.html, htroot/CrawlStartSite.html, htroot/js/IndexCreate.js |
Thu Jan 16 19:54:33 CET 2020 by Michael Peter Christen | removed old jsoup lib from eclipse classpath Changed Files: .classpath |
Wed Jan 15 15:24:36 CET 2020 by Michael Christen | new servlet path "p2p" with a test class. Call the class with http://localhost:8090/p2p/seeds.json Changed Files: .classpath, htroot/p2p/seeds.java, htroot/p2p/seeds.json |
Sun Dec 29 01:21:39 CET 2019 by Michael Peter Christen | scanner crawl starts wants non-parseable files Changed Files: htroot/CrawlStartScanner_p.java |
Sat Dec 14 21:02:10 CET 2019 by reger | upd to jsoup-1.12.1.jar Changed Files: build.xml, lib/jsoup-1.12.1.jar, pom.xml |
Tue Nov 05 14:44:01 CET 2019 by Michael Peter Christen | masking connects Changed Files: htroot/Network.json |
Tue Nov 05 11:50:56 CET 2019 by Michael Peter Christen | grand CORS for json files Changed Files: source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/server/http/HTTPDProxyHandler.java, source/net/yacy/server/http/HTTPDemon.java |
Tue Nov 05 11:36:23 CET 2019 by Michael Peter Christen | typo Changed Files: htroot/Network.json |
Tue Nov 05 11:29:07 CET 2019 by Michael Peter Christen | added json version of peer list Changed Files: htroot/Network.json |
Thu Oct 17 01:35:44 CEST 2019 by Michael Christen | removed donation message in search result column Changed Files: defaults/yacy.init, source/net/yacy/yacy.java |
Mon Oct 14 03:39:44 CEST 2019 by Michael Christen | preliminary installerfinished link Changed Files: build.nsi |
Mon Oct 14 01:36:33 CEST 2019 by Michael Christen | going back to the old links - without https Changed Files: build.nsi |
Mon Oct 14 01:02:41 CEST 2019 by Michael Christen | NSISdl does not support the HTTPS protocol. lets try the http link for the linux download link.. see also: https://stackoverflow.com/questions/46467921/download-of-file-during-nsis-installer-run-freezes-at-connecting Changed Files: build.nsi |
Sun Oct 13 11:02:19 CEST 2019 by Michael Peter Christen | trying to fix 'Expected feature release number in range of 9 to 14, but got: 8' error while travis is building Changed Files: .travis.yml |
Sun Oct 13 10:53:03 CEST 2019 by Michael Peter Christen | code freeze for release Changed Files: build.properties |
Sun Sep 29 00:26:48 CEST 2019 by Michael Peter Christen | more space for sponsoring Changed Files: defaults/yacy.init |
Sat Sep 28 23:16:23 CEST 2019 by Michael Peter Christen | switched url and snippet position Changed Files: htroot/yacysearchitem.html, skins/pdblue.css |
Sat Sep 28 22:11:11 CEST 2019 by Michael Peter Christen | enhanced search result design Changed Files: htroot/yacysearch.html, skins/pdblue.css |
Sat Sep 28 21:27:22 CEST 2019 by Michael Christen | added sponsor link Changed Files: htroot/env/templates/header.template |
Sat Sep 28 02:07:08 CEST 2019 by Michael Christen | removed some warnings Changed Files: source/net/yacy/cora/protocol/Domains.java, source/net/yacy/server/serverSwitch.java |
Sat Sep 28 02:06:50 CEST 2019 by Michael Christen | design updates + added link to new forum Changed Files: htroot/env/grafics/lock.gif, htroot/env/grafics/lockopen.gif, htroot/env/templates/header.template, skins/pdblue.css |
Thu Sep 26 23:44:04 CEST 2019 by Michael Peter Christen | design changes - more space Changed Files: htroot/env/base.css, htroot/env/bootstrap-base.css, htroot/env/bootstrap/css/bootstrap.css, htroot/yacysearch.html, skins/pdblue.css |
Wed Sep 25 22:51:25 CEST 2019 by Michael Peter Christen | updated seedlist bootstrap locations Changed Files: defaults/yacy.network.freeworld.unit |
Wed Sep 25 16:49:50 CEST 2019 by Michael Christen | added patreon Changed Files: .github/FUNDING.yml |
Wed Sep 25 15:59:05 CEST 2019 by Michael Christen | added GitHub Sponsors tag Changed Files: .github/FUNDING.yml |
Wed Jul 17 09:36:45 CEST 2019 by Michael Christen | Merge pull request #303 from arankaren/master Update spanish language Changed Files: locales/es.lng, locales/master.lng.xlf |
Mon Jul 15 22:20:58 CEST 2019 by Arankaren Irene | removed comment line Changed Files: locales/es.lng |
Mon Jul 15 22:07:42 CEST 2019 by AranKaren Irene | update spanish language Changed Files: locales/es.lng |
Mon Jul 15 22:07:18 CEST 2019 by AranKaren Irene | fix syntax Changed Files: locales/master.lng.xlf |
Sat Jun 15 11:25:12 CEST 2019 by Michael Christen | updated compilation with eclipse and links to forum Changed Files: README.md |
Thu May 23 11:36:40 CEST 2019 by Michael Christen | added github funding Changed Files: .github/FUNDING.yml |
Wed May 01 08:54:19 CEST 2019 by luccioman | New optional crawl filter on the URL a doc must match to crawl its links For finer control over which parsed documents can trigger an addition of their links to the crawl stack, complementary to the existing crawl depth parameter. Changed Files: htroot/CrawlProfileEditor_p.xml, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/search/Switchboard.java |
Fri Apr 19 10:33:58 CEST 2019 by luccioman | Upgraded PDFBox dependency from 2.0.14 to 2.0.15 Changed Files: .classpath, build.xml, lib/fontbox-2.0.15.License, lib/fontbox-2.0.15.jar, lib/pdfbox-2.0.15.License, lib/pdfbox-2.0.15.jar, pom.xml |
Thu Apr 18 15:31:38 CEST 2019 by luccioman | Added "Show Metadata" links to the ViewFile.html links mode To conveniently follow parsed links in the file viewer Changed Files: htroot/ViewFile.html, htroot/ViewFile.java, source/net/yacy/cora/document/encoding/UTF8.java |
Thu Apr 18 14:23:00 CEST 2019 by luccioman | Better HTML text semantics for technical descriptions Changed Files: htroot/Settings_HttpClient.inc |
Fri Apr 05 11:06:09 CEST 2019 by luccioman | Render a relevant message and status on blocked search requests When unauthenticated (or with insufficient rights) client is blocked either because blacklisted or excessive request rate, render an error message and a relevant HTTP status for API requests, instead of an empty response that appears broken. Changed Files: htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java |
Wed Apr 03 14:21:53 CEST 2019 by luccioman | Allow JS resorting of search results by unauthenticated users Acces rate limitations to this search mode by unauthenticated users are set low by default to prevent unwanted server overload but can be customized through the SearchAccessRate_p.html configuration page Fixes #291 Changed Files: defaults/yacy.init, htroot/ConfigPortal_p.html, htroot/SearchAccessRate_p.html, htroot/yacysearch.java, locales/zh.lng, source/net/yacy/search/SearchAccessRateConstants.java |
Wed Mar 27 18:23:41 CET 2019 by luccioman | Added Solr GSA writer support for responses from remote instances Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java |
Fri Mar 22 09:52:57 CET 2019 by luccioman | Upgraded PDFBox dependency from 2.0.11 to 2.0.14 Changed Files: .classpath, build.xml, lib/fontbox-2.0.14.License, lib/fontbox-2.0.14.jar, lib/pdfbox-2.0.14.License, lib/pdfbox-2.0.14.jar, pom.xml |
Fri Mar 22 09:50:30 CET 2019 by luccioman | Properly close PDF snapshots loaded with pdfbox library Changed Files: source/net/yacy/cora/util/Html2Image.java |
Wed Mar 20 18:24:16 CET 2019 by luccioman | Added Solr GrepHTML writer support for responses from remote instances Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GrepHTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Tue Mar 19 11:25:44 CET 2019 by luccioman | Added Solr snapshots writer support for responses from remote instances Changed Files: source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java |
Sat Mar 16 10:19:47 CET 2019 by luccioman | Merge pull request #283 from tangdou1/patch-1 Update zh.lng Changed Files: locales/zh.lng |
Fri Mar 15 17:46:23 CET 2019 by luccioman | P2P/Privacy switch buttons support with JavaScript disabled Changed Files: htroot/env/base.css, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/zh.lng |
Sat Mar 09 10:21:48 CET 2019 by luccioman | Support for search result switch buttons with JavaScript disabled Changed Files: htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java |
Sat Mar 09 08:28:39 CET 2019 by luccioman | Properly render the href attribute of the active page button Changed Files: htroot/yacysearchpagination.html |
Tue Mar 05 08:18:18 CET 2019 by luccioman | Properly fill current page number to new server side pagination template When current page is automatically reset to zero because of a new search request. Changed Files: htroot/yacysearch.html, htroot/yacysearchpagination.java |
Mon Mar 04 10:05:47 CET 2019 by luccioman | Improve accuracy of total results count on latest pages in Stealth mode Previously, when mixing results from local RWI and local Solr (Stealth mode), total local Solr count could be ignored on last result pages, when the page offset was higher than local Solr count but lower than total RWI count. Changed Files: source/net/yacy/peers/Protocol.java |
Thu Feb 28 22:56:49 CET 2019 by luccioman | Server side initial pagination links rendering For better support of the search page usage with JavaScript disabled. Reduces also the number of initial refreshes of the paginations links. When JavaScript is enabled, pagination links are still regularly refreshed until all the search feeds are terminated on server side. Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearchpagination.html, htroot/yacysearchpagination.java |
Tue Feb 26 10:55:27 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Tue Feb 26 10:39:25 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Tue Feb 19 10:54:26 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Tue Feb 19 10:35:51 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Tue Feb 19 10:21:55 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Wed Feb 13 22:13:58 CET 2019 by luccioman | Merge pull request #278 from tangdou1/master Update the chinese translation Changed Files: locales/zh.lng |
Fri Feb 08 12:23:01 CET 2019 by luccioman | Render additional embedded audios from links on extended audio search Changed Files: htroot/env/base.css, htroot/js/yacysearchAudio.js, htroot/yacysearch.html, htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Wed Feb 06 04:46:16 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Wed Feb 06 04:38:14 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Tue Feb 05 09:43:28 CET 2019 by luccioman | Relevant message when using the stop script while YaCy is not running Changed Files: stopYACY.sh |
Mon Feb 04 14:34:37 CET 2019 by luccioman | Added a stop command using the existing stop script to the snap package Changed Files: addon/snap/exportYacyDataPath.sh, addon/snap/snapPasswd.sh, addon/snap/snapStopYACY.sh, snap/snapcraft.yaml, stopYACY.sh |
Sun Feb 03 01:42:30 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Sat Feb 02 10:49:36 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Sat Feb 02 09:59:24 CET 2019 by luccioman | Improved logging for federated search - Do not use spaces in logger identifier name so the log level can be configured in yacy.logging - Hold the logger instance to avoid the logging system to look for it from its name at each appended log message Changed Files: source/net/yacy/cora/federate/AbstractFederateSearchConnector.java, source/net/yacy/cora/federate/FederateSearchManager.java |
Sat Feb 02 09:26:16 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Sat Feb 02 00:42:37 CET 2019 by luccioman | Removed no more available OpenSearch URL example Changed Files: defaults/heuristicopensearch.conf |
Sat Feb 02 00:41:49 CET 2019 by luccioman | Updated federated search html results mapping example Changed Files: defaults/federatecfg/npmjs.html.map.properties |
Fri Feb 01 04:29:57 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Fri Feb 01 04:26:35 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Fri Feb 01 03:53:57 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Thu Jan 31 10:18:41 CET 2019 by luccioman | Set Snap source type to git for easier rebuild from main master sources Also changed grade to allow promoting YaCy snap to a channel with higher level than beta. Changed Files: snap/snapcraft.yaml |
Thu Jan 31 04:57:41 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Thu Jan 31 03:20:11 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Thu Jan 31 02:31:40 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Wed Jan 30 19:09:21 CET 2019 by luccioman | Added Readme links to Gitter and Matrix rooms Fixes issue #268 Changed Files: README.md |
Wed Jan 30 18:43:13 CET 2019 by luccioman | Added basic controls to play all audio results. Not displayed when JavaScript is disabled. Changed Files: htroot/env/base.css, htroot/js/yacysearch.js, htroot/js/yacysearchAudio.js, htroot/jslicense.html, htroot/yacysearch.html, htroot/yacysearchitem.html |
Wed Jan 30 14:41:46 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Wed Jan 30 11:06:36 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Wed Jan 30 10:40:34 CET 2019 by tangdou1 | Merge pull request #1 from tangdou1/patch-1 Resort and update the translation in zh.lng according to the master.lng.xlf Changed Files: locales/zh.lng |
Wed Jan 30 10:38:22 CET 2019 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Sat Jan 26 10:28:44 CET 2019 by luccioman | Fixed shell start script failure on missing memory settings in conf file Before that fix, the startYACY.sh failed when the javastart_Xmx or javastart_Xms configuration keys were missing for some reason from yacy.conf file, with the error "Unrecognized option: - Error: Could not create the Java Virtual Machine." Changed Files: startYACY.sh |
Sat Jan 19 11:29:17 CET 2019 by luccioman | Upgraded icu4j dependency from 62.1 to 63.1 Changed Files: .classpath, build.xml, lib/icu4j-63.1.jar, pom.xml |
Mon Jan 14 11:08:11 CET 2019 by luccioman | Properly read configured memory settings when using a custom data path Fixes issue #269 Changed Files: startYACY.sh |
Thu Jan 10 20:02:57 CET 2019 by sgaebel | remove warnings: 'Statement unnecessarily nested within else clause' Changed Files: htroot/TransNews_p.java, htroot/Translator_p.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/document/importer/WarcImporter.java, source/net/yacy/search/navigator/LanguageNavigator.java, source/net/yacy/search/navigator/YearNavigator.java, source/net/yacy/utils/translation/TranslationManager.java, source/net/yacy/utils/translation/TranslatorXliff.java |
Tue Jan 08 20:45:59 CET 2019 by sgaebel | bump to HTTPclient-4.5.6 Changed Files: .classpath, build.xml, lib/httpclient-4.5.6.License, lib/httpclient-4.5.6.jar, lib/httpcore-4.4.10.License, lib/httpcore-4.4.10.jar, lib/httpmime-4.5.6.License, lib/httpmime-4.5.6.jar, pom.xml |
Sat Jan 05 19:30:52 CET 2019 by luccioman | Added Snap package configuration to choose to version data or not By default when the Snap package is installed, YaCy data is stored in a versioned user folder, allowing to revert to previous data after a package refresh for example. But it can consumes much disk space, so it is now possible to tell YaCy snap not to version its data, with the Snap configuration setting "data.versioned=false". Changed Files: addon/snap/snapPasswd.sh, addon/snap/snapStartYACY.sh, snap/hooks/configure, snap/snapcraft.yaml |
Fri Jan 04 18:58:49 CET 2019 by sgaebel | taking care of closing inputstreams, HTTPClient Changed Files: source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/xml/opensearchdescriptionReader.java |
Fri Jan 04 18:46:59 CET 2019 by sgaebel | Recrawl: - set the chunksize to 100 to meet the max of the embedded solr - re-enable sorting (the case where we switched it of should be away) - enable recrawling on remote-solr Changed Files: htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Fri Jan 04 18:27:44 CET 2019 by sgaebel | extend the SolrServlet to be usable as remote solr (incl. update) this feature needs to be enabled by uncomment the url-pattern Changed Files: defaults/web.xml, source/net/yacy/http/servlets/SolrServlet.java |
Sat Dec 22 14:42:22 CET 2018 by luccioman | Removed usage of now deprecated Jetty function Changed Files: source/net/yacy/http/InetPathAccessHandler.java |
Thu Dec 20 15:02:07 CET 2018 by luccioman | Cleaned up Spanish translation after merge of PR #238 * Fixed some indentation * Removed untranslated entries Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, locales/es.lng |
Thu Dec 20 14:57:14 CET 2018 by luccioman | Merge pull request #238 from ivanhercaz/esLang [WIP] Spanish translation Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, locales/es.lng |
Thu Dec 20 14:53:01 CET 2018 by luccioman | Exclude peers with empty or disabled RWI from remote RWI search Changed Files: source/net/yacy/peers/DHTSelection.java, source/net/yacy/peers/RemoteSearch.java |
Mon Dec 17 10:34:51 CET 2018 by luccioman | Fixed wkhtmltopdf integration in Alpine flavor Docker image Building now wkhtmltopdf (used for snapshots generation) from sources, as its package is only available on the Alpine edge branch and is not compatible with the current Alpine (3.8) stable base image used for YaCy. Changed Files: docker/Dockerfile.alpine |
Fri Dec 14 21:07:06 CET 2018 by luccioman | Fixed Alpine flavor Docker image (wkhtmltopdf moved to community repo) See also https://github.com/alpinelinux/aports/commit/10a2b3a8cd75ab13660bef4a97f351e031fba917 Changed Files: docker/Dockerfile.alpine |
Wed Dec 12 08:43:26 CET 2018 by luccioman | Added support for pdf snapshots rendering to Snap package Changed Files: snap/snapcraft.yaml |
Tue Dec 11 22:31:31 CET 2018 by luccioman | Added a configurable timeout to wkhtmltopdf calls for pdf snapshots Necessary to prevent blocking the indexing workflow when some wkhtmltopdf renderings fail without terminating Changed Files: defaults/yacy.init, source/net/yacy/cora/util/Html2Image.java, source/net/yacy/crawler/data/Transactions.java, source/net/yacy/kelondro/util/OS.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Mon Dec 10 17:53:12 CET 2018 by luccioman | Updated up.sh to support passing the target peer address as a parameter Changed Files: bin/up.sh |
Sat Dec 08 15:49:55 CET 2018 by luccioman | Merge pull request #261 from agnelvishal/patch-3 Update searchall1.sh This should have been done ealrier after removal of up1.sh script in commit 3d9ce9cd04014fa798f6b2514cf89029675651f0. Changed Files: bin/searchall1.sh |
Sat Dec 08 15:33:56 CET 2018 by luccioman | Merge pull request #260 from agnelvishal/patch-2 Update up.sh It looks like this is a missing impact for earlier commit 65e6ae52fb6df83920b6bb667a4486c0ca67e532. Changed Files: bin/up.sh |
Fri Dec 07 21:15:05 CET 2018 by Agnel Vishal | Grammar change (#258) Grammar change to Readme APIs section Changed Files: README.md |
Thu Dec 06 20:18:00 CET 2018 by luccioman | Properly resolve relative URLs against document URL in html base tags Fixes issue #256 Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java |
Thu Dec 06 11:39:10 CET 2018 by Agnel Vishal | Update searchall1.sh The up1.sh does not exist. up.sh does the work Changed Files: bin/searchall1.sh |
Thu Dec 06 11:37:38 CET 2018 by Agnel Vishal | Update up.sh IP addresses are not selected if "&ip=" is present Changed Files: bin/up.sh |
Thu Dec 06 09:53:08 CET 2018 by luccioman | Extended detection of external tools used for Snapshots generation This enable detecting wkhtmltopdf and Imagemagick convert executables when they are at system Path in addition to common installation paths. Changed Files: source/net/yacy/cora/util/Html2Image.java |
Sat Dec 01 10:59:13 CET 2018 by luccioman | Added a command to set the admin password from the Snap package Fixes issue #254 Changed Files: snap/snapcraft.yaml |
Wed Nov 28 21:44:49 CET 2018 by luccioman | Let setting admin password with script when YaCy server is not running Still using the ConfigAccounts_p API when the server is running. Changed Files: bin/passwd.sh |
Tue Nov 27 11:41:00 CET 2018 by luccioman | Improvements to passwd.sh script - Use the configured administrator user name instead of always defaulting to "admin" - Do not echo the password in clear text - Check the password minimum size as will be applied in ConfigAccounts_p - Let user type a password when not provided as a parameter Changed Files: bin/passwd.sh |
Tue Nov 27 11:36:40 CET 2018 by luccioman | Also check yacy.conf file existence in shell scripts where relevant Changed Files: bin/apicall.sh, bin/apicat.sh, bin/checkConfFile.sh, bin/down.sh, bin/graphicstest.sh, bin/protectedPostApiCall.sh, bin/search.sh, bin/searchall.sh |
Wed Nov 21 17:59:43 CET 2018 by luccioman | Updated the JRE URL from 8u181 to 8u191 for the MS Windows installer Changed Files: build.nsi |
Mon Nov 19 08:49:13 CET 2018 by luccioman | Enable soft autocommit in default Solr config Since upgrade from Solr 5.5 to Solr 6.6 (commit 6fe7359), hard autocommits were still enabled to regularly persist the Solr index to the file system, but new index entries were no more automatically made available for use by the application (soft autocommit). Therefore, YaCy features such as index statistics, that do not perform an explicit commit (as recommended by Solr documentation) were no more accurate. Soft autocommit is now restored as a default, with a time period expected to be sufficient for accuracy while adding only a reasonable system load overhead. Fixes issue #251 Changed Files: defaults/solr/solrconfig.xml |
Mon Nov 19 08:40:17 CET 2018 by luccioman | Updated mirror repository source url used in alpine flavor Docker image Switching to a mirror providing a valid certificate Changed Files: docker/Dockerfile.alpine |
Thu Nov 15 17:32:22 CET 2018 by luccioman | Trace a message on incomplete proper server finish when killing process Changed Files: source/net/yacy/yacy.java |
Tue Nov 13 17:50:29 CET 2018 by luccioman | Added a link to YaCy Gitter chat room in Readme As suggested by PR #159 and reminded by @ivanhercaz in issue #247 Changed Files: README.md |
Tue Nov 13 07:50:27 CET 2018 by luccioman | Added details to the keystore configuration properties documentation Changed Files: defaults/yacy.init |
Sat Nov 10 22:00:49 CET 2018 by luccioman | Fixed concurrency issue on cache used for circles rendering Without synchronization lock, concurrent rendering of images including circles could lead to glitches as reported in issue #248 Changed Files: source/net/yacy/visualization/CircleTool.java, test/java/net/yacy/visualization/CircleToolTest.java |
Thu Nov 08 14:42:40 CET 2018 by Michael Peter Christen | evil bots may crawl harder Changed Files: source/net/yacy/cora/protocol/ClientIdentification.java |
Thu Nov 08 08:39:32 CET 2018 by luccioman | Merge pull request #250 from theel0ja/patch-1 Improved formatting of markdown Changed Files: README.md |
Wed Nov 07 14:52:42 CET 2018 by luccioman | Fixed gzip decompression regression on index transfer APIs Processing of gzip encoded incoming requests (on /yacy/transferRWI.html and /yacy/transferURL.html) was no more working since upgrade to Jetty 9.4.12 (see commit 51f4be1). To prevent any conflicting behavior with Jetty internals, use now the GzipHandler provided by Jetty to decompress incoming gzip encoded requests rather than the previously used custom GZIPRequestWrapper. Fixes issue #249 Changed Files: source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Tue Nov 06 10:10:09 CET 2018 by luccioman | Fixed termination of Host browser and link structure Solr query threads On some conditions (especially when reaching timeout), concurrent Solr query tasks used by the /HostBrowser.html and /api/linkstructure.json never terminated, thus leaking resources, as reported by @Vort in issue #246 Changed Files: htroot/HostBrowser.java, source/net/yacy/cora/federate/solr/connector/AbstractSolrConnector.java, source/net/yacy/cora/federate/solr/connector/SolrConnector.java, source/net/yacy/search/schema/HyperlinkGraph.java |
Mon Nov 05 08:27:17 CET 2018 by Elias Ojala | Improved formatting of markdown Changed Files: README.md |
Wed Oct 31 07:43:42 CET 2018 by luccioman | Limit length of initially visible text in link structure graph nodes To improve a bit readability of graphs having a large number of nodes. Changed Files: htroot/env/hypertree.css, htroot/js/hypertree.js |
Sun Oct 28 10:07:46 CET 2018 by luccioman | Upgraded d3js dependency from 3.4.4 to 5.7.0 Changed Files: htroot/Crawler_p.html, htroot/HostBrowser.html, htroot/js/d3.v5.min.js, htroot/js/hypertree.js, htroot/jslicense.html |
Fri Oct 26 09:11:23 CEST 2018 by luccioman | Added missing CSS width units to some HostBrowser.html styling Changed Files: htroot/HostBrowser.html |
Fri Oct 26 08:55:25 CEST 2018 by luccioman | Fixed "TypeError: table.tBodies[0] is undefined" host browser JS error Traced in browser console when a host details table is empty. Changed Files: htroot/HostBrowser.html |
Fri Oct 19 10:12:10 CEST 2018 by luccioman | Render http status instead of null responses on snapshot api errors Changed Files: htroot/api/snapshot.java |
Thu Oct 18 12:41:57 CEST 2018 by luccioman | Added suport for PDF snapshots generation when running on MS Windows Changed Files: source/net/yacy/cora/protocol/ClientIdentification.java, source/net/yacy/cora/util/Html2Image.java |
Wed Oct 17 10:04:48 CEST 2018 by luccioman | Upgraded Apache Ant from 1.10.1 to 1.10.5 in Docker alpine image flavor Changed Files: docker/Dockerfile.alpine |
Wed Oct 17 08:42:31 CEST 2018 by luccioman | Added support for snapshots generation to Docker images Changed Files: docker/Dockerfile, docker/Dockerfile.alpine, docker/Readme.md |
Tue Oct 16 08:02:43 CEST 2018 by luccioman | Added a crawl start hint message on availability or not of wkhtmltopdf As this tool is required to produce pdf snapshots Changed Files: htroot/CrawlStartExpert.html |
Mon Oct 15 18:19:57 CEST 2018 by luccioman | Added basic styling for cleaner rendering of missing image snapshots For the output of the Solr snapshots writer Changed Files: htroot/env/base.css, source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java |
Sun Oct 14 10:30:30 CEST 2018 by luccioman | Render a relevant HTTP status code on snapshot image rendering error Instead of a null response body which is not very helpful. Changed Files: htroot/api/snapshot.java, source/net/yacy/http/servlets/TemplateMissingParameterException.java, source/net/yacy/http/servlets/TemplateProcessingException.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Sat Oct 13 10:36:39 CEST 2018 by luccioman | Updated Solr snapshots writer for a cleaner html head Changed Files: source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java |
Sat Oct 13 10:22:47 CEST 2018 by luccioman | Made snapshots size customizable in Solr snapshots response writer Changed Files: source/net/yacy/cora/federate/solr/responsewriter/SnapshotImagesReponseWriter.java |
Thu Oct 11 18:34:39 CEST 2018 by luccioman | Fixed concurrency issue in custom classloader used for template classes As reported in issue #241, the problem is only critical (random but complete crash of the JVM) when upgrading to JDK11. Changed Files: source/net/yacy/server/serverClassLoader.java |
Mon Oct 08 16:50:02 CEST 2018 by luccioman | Fixed remaining blacklist entries improper decoding of '+' character In the blacklist cleaner and import/export administration pages. Changed Files: htroot/BlacklistCleaner_p.java, htroot/sharedBlacklist_p.java |
Thu Oct 04 09:33:58 CEST 2018 by luccioman | Decode blacklist entries for easier edition of non ascii chars Not using the JDK URLDecoder.decode() function, as it strips '+' characters when they occur after '?' (both characters having regular expression semantics when used in blacklist path patterns) Changed Files: htroot/Blacklist_p.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/repository/Blacklist.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Tue Oct 02 14:36:13 CEST 2018 by luccioman | Improved normalization of blacklist path patterns having non ascii chars Normalize blacklist path patterns using percent-encoding, at pattern edition in web interface and at loading from configuration files. Fixes issue #237 Changed Files: htroot/Blacklist_p.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/BlacklistHelper.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Tue Oct 02 07:27:47 CEST 2018 by luccioman | Merge pull request #240 from ivanhercaz/fixEnglishBookmarksPage Fix English Bookmarks.html Changed Files: htroot/Bookmarks.html |
Mon Oct 01 01:37:08 CEST 2018 by ivanhercaz | typo fix Changed Files: locales/es.lng |
Mon Oct 01 01:36:09 CEST 2018 by ivanhercaz | cleaning the file of entries in German already translated to Spanish Changed Files: locales/es.lng |
Mon Oct 01 01:28:58 CEST 2018 by ivanhercaz | changin all «» to "" to avoid confusions Changed Files: locales/es.lng |
Mon Oct 01 01:22:08 CEST 2018 by ivanhercaz | ConfigHeuristics_p.html translated Changed Files: locales/es.lng |
Sun Sep 30 15:33:09 CEST 2018 by ivanhercaz | ConfigHTCache_p.html translated Changed Files: locales/es.lng |
Sun Sep 30 03:10:42 CEST 2018 by ivanhercaz | adding Spanish to the interface language list Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java |
Sun Sep 30 03:00:06 CEST 2018 by ivanhercaz | ConfigAccounts_p.html translated Changed Files: locales/es.lng |
Sun Sep 30 02:37:35 CEST 2018 by ivanhercaz | Collage.html translated Changed Files: locales/es.lng |
Sun Sep 30 02:16:56 CEST 2018 by ivanhercaz | Bookmarks.html translated Changed Files: locales/es.lng |
Sun Sep 30 02:00:43 CEST 2018 by ivanhercaz | typo fix in Bookmarks.html Changed Files: htroot/Bookmarks.html |
Fri Sep 28 16:06:01 CEST 2018 by ivanhercaz | removing duplicated entry (the one in German) for Translator_p.html Changed Files: locales/es.lng |
Fri Sep 28 16:04:38 CEST 2018 by ivanhercaz | BlogComments.html translated Changed Files: locales/es.lng |
Fri Sep 28 15:45:37 CEST 2018 by ivanhercaz | Translator_p.html translated Changed Files: locales/es.lng |
Fri Sep 28 01:30:54 CEST 2018 by ivanhercaz | Blog.html translated Changed Files: locales/es.lng |
Fri Sep 28 00:54:56 CEST 2018 by ivanhercaz | Blacklist_p.html translated Changed Files: locales/es.lng |
Thu Sep 27 21:21:24 CEST 2018 by ivanhercaz | BlacklistTest_p.html translated and forgotten string in BlacklistImpExp_p.html Changed Files: locales/es.lng |
Thu Sep 27 21:10:17 CEST 2018 by ivanhercaz | BlacklistImpExp_p.html translated Changed Files: locales/es.lng |
Thu Sep 27 20:04:50 CEST 2018 by ivanhercaz | BlacklistCleaner_p.html translated Changed Files: locales/es.lng |
Thu Sep 27 19:33:41 CEST 2018 by ivanhercaz | Autocrawl_p.html translated Changed Files: locales/es.lng |
Thu Sep 27 14:16:46 CEST 2018 by ivanhercaz | AccessTracker_p.html translated Changed Files: locales/es.lng |
Thu Sep 27 11:11:50 CEST 2018 by ivanhercaz | AccessGrid_p.html translated Changed Files: locales/es.lng |
Wed Sep 26 08:47:13 CEST 2018 by luccioman | Removed now unused Java import in addition to modification from PR #239 Changed Files: htroot/IndexCreateParserErrors_p.java |
Wed Sep 26 08:43:48 CEST 2018 by luccioman | Merge pull request #239 from otteresk/master Display correct time in Rejected URLs overview Changed Files: htroot/IndexCreateParserErrors_p.java |
Tue Sep 25 20:43:58 CEST 2018 by otter | replace current date by FailDate Changed Files: htroot/IndexCreateParserErrors_p.java |
Tue Sep 25 02:45:59 CEST 2018 by ivanhercaz | starting the Spanish translation Changed Files: locales/es.lng |
Fri Sep 21 12:07:57 CEST 2018 by luccioman | Upgraded BouncyCastle dependencies from jdk15:1.46 to jdk15on:1.60 Changed Files: .classpath, build.xml, lib/bcmail-jdk15on-1.60.License, lib/bcmail-jdk15on-1.60.jar, lib/bcpkix-jdk15on-1.60.License, lib/bcpkix-jdk15on-1.60.jar, lib/bcprov-jdk15on-1.60.License, lib/bcprov-jdk15on-1.60.jar, pom.xml |
Fri Sep 21 09:59:32 CEST 2018 by luccioman | Hide password values from visible HTML in the Advanced Config page Fixes issue #228 Changed Files: htroot/ConfigProperties_p.html, htroot/ConfigProperties_p.java |
Wed Sep 19 09:32:59 CEST 2018 by luccioman | Use accessible labels in the Server Access Settings page Changed Files: htroot/Settings_ServerAccess.inc |
Mon Sep 17 08:07:27 CEST 2018 by luccioman | Updated html input field type for seed upload with file method - To meet current browsers security rules, which prevent selecting a full file path with an html input field of type 'file' - As it does not make sense to select a local file path when a the administered YaCy server is remote (not on the same computer as the browser) Changed Files: htroot/Settings_Seed_UploadFile.inc |
Sun Sep 16 10:37:47 CEST 2018 by luccioman | Use a constant default target file name for seed SCP upload method To make seed upload (in /Settings_p.html?page=seed page) with SCP easier when the user specify a remote target directory path. See report by @vikulin in issue #227 Changed Files: source/net/yacy/peers/operation/yacySeedUploadScp.java |
Fri Sep 14 14:02:32 CEST 2018 by luccioman | Removed some dead code Changed Files: source/net/yacy/http/servlets/UrlProxyServlet.java |
Thu Sep 13 12:17:02 CEST 2018 by luccioman | Improved support for Java logs file pattern options - support of "%h" and "%t" pattern components - more proper initialization of file handler when the data folder is not the default one, notably to prevent a non blocking but ugly error stack trace reported by the log manager at startup with that kind of setup Changed Files: source/net/yacy/cora/util/ConcurrentLog.java, source/net/yacy/yacy.java |
Wed Sep 12 17:34:40 CEST 2018 by luccioman | Fixed raw IPV6 addresses snapshots read/write on FAT32 and NTFS fs Fixes issue #225 Changed Files: source/net/yacy/crawler/data/Snapshots.java, source/net/yacy/crawler/data/Transactions.java |
Wed Sep 12 07:29:25 CEST 2018 by luccioman | Fixed rendering of the YMarks.html page Also to clarify which pages still depends on old JQuery and JQuery UI dependencies. Changed Files: htroot/YMarks.html |
Tue Sep 11 11:28:59 CEST 2018 by luccioman | Merge pull request #218 from edom/master Add contributor guidelines; closes #214 Changed Files: CONTRIBUTING.md |
Tue Sep 11 11:28:31 CEST 2018 by luccioman | Merge pull request #226 from luccioman/master Snapcraft packaging adjustements Changed Files: addon/YaCy256x256.png, snap/snapcraft.yaml |
Mon Sep 10 22:14:34 CEST 2018 by luccioman | Added an icon matching Snapcraft packaging icon size requirements Changed Files: addon/YaCy256x256.png, snap/snapcraft.yaml |
Fri Aug 31 10:14:27 CEST 2018 by luccioman | Renamed Snap with a name detected as available on the Snap store Changed Files: snap/snapcraft.yaml |
Fri Aug 31 08:34:28 CEST 2018 by luccioman | Fixed logs folder checking and creation Previously, if YaCy log folder was for example at `/home/user/yacy/DATA/LOG`, because of improper truncation of log path, an unnecessary directory creation was atempted at `/home/us`. Changed Files: source/net/yacy/cora/util/ConcurrentLog.java |
Thu Aug 30 18:21:14 CEST 2018 by luccioman | Added a first version of Snapcraft configuration file for Snap packaging Changed Files: snap/snapcraft.yaml |
Thu Aug 30 18:16:22 CEST 2018 by luccioman | Made possible to provide an absolute data root path for start script Previously, only a path relative to the user home folder could be provided Changed Files: source/net/yacy/yacy.java, startYACY.sh |
Tue Aug 28 15:49:30 CEST 2018 by luccioman | Fetch result pages one by one when scrolling in portal search widget To prevent unnecessary load and items retrival errors on backend Changed Files: htroot/portalsearch/yacy-portalsearch.js |
Tue Aug 28 14:59:25 CEST 2018 by luccioman | Configured local peer as default portal search widget backend Rather than relying on a peer eventually deployed on search.yacy.net Changed Files: htroot/portalsearch/yacy-portalsearch.html |
Tue Aug 28 14:40:57 CEST 2018 by luccioman | Removed implicit global JavaScript variables from portal search widget Changed Files: htroot/portalsearch/yacy-portalsearch.js |
Tue Aug 28 11:39:16 CEST 2018 by luccioman | Limit search API calls rate when typing in the search portal widget Changed Files: htroot/portalsearch/yacy-portalsearch.js |
Tue Aug 28 08:34:23 CEST 2018 by luccioman | Replaced RWI ranking JQuery sliders with standard HTML range inputs Considering that the sliders usage on that page is very basic, using standard HTML5 inputs of type "range" has here the following advantages : - better keyboard accessibility - remove not very necessary additional jquery dependencies Today browsers suport for range inputs is good, and even on old unsupporting browsers such as IE < 10 they nicely fall back to text inputs. Changed Files: htroot/RankingRWI_p.html |
Sat Aug 25 10:43:38 CEST 2018 by luccioman | Merge pull request #220 from pnhofmann/master Improve Dockerfile Changed Files: docker/Dockerfile, docker/Dockerfile.alpine, docker/Readme.md |
Fri Aug 24 09:13:12 CEST 2018 by luccioman | Added possibility to hide or show image results with rendering errors When searching images, thumbnails that could not be rendered (because of a load error such as HTTP 404, networking issue or an internal error on the rendering servlet) are now hidden as default. But can be revealed with a button if desired. Fix for issue #217 Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearchitem.html |
Mon Aug 20 16:38:51 CEST 2018 by Philipp Hofmann | Docker: merge RUN instructions for less layer (-3) Changed Files: docker/Dockerfile, docker/Dockerfile.alpine |
Wed Aug 22 22:35:00 CEST 2018 by luccioman | Removed deprecated warning comments about imports and Debian installer Deprecated by commit be5d3a106677006995e309e1ce587c731f0cabf0 , as classpath is now defined in yacycore.jar Manifest file. Changed Files: source/net/yacy/cora/order/Base64Order.java, source/net/yacy/cora/order/Digest.java |
Mon Aug 20 17:01:08 CEST 2018 by luccioman | Fixed encoding of '+' character on search pages links As revealed by issue #216 Changed Files: htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template, htroot/index.html, htroot/yacysearchitem.java, htroot/yacysearchtrailer.java, source/net/yacy/server/serverObjects.java |
Mon Aug 20 14:52:01 CEST 2018 by Philipp Hofmann | Remove executable (x) permission of 2 files * docker/Dockerfile.alpine * docker/Readme.md Changed Files: docker/Dockerfile.alpine, docker/Readme.md |
Mon Aug 20 14:42:20 CEST 2018 by Philipp Hofmann | Dockerfile: Improve package cache update * Alpine-Image: If --no-cache is used, apk update is not necessary * Debian-Image: Remove /var/lib/apt/lists to reduce image size Changed Files: docker/Dockerfile, docker/Dockerfile.alpine |
Mon Aug 20 14:43:05 CEST 2018 by Philipp Hofmann | Dockerfile: Remove bad whitespaces Changed Files: docker/Dockerfile, docker/Dockerfile.alpine |
Mon Aug 20 08:10:39 CEST 2018 by luccioman | Removed unnecessary '+' character URL decoding from search query Manually replacing '+' character or "%20" by a space character in the search query parameter was necessary in YaCy a long time ago to properly decode application/x-www-form-urlencoded format (commit 9842fab6e47f1dd93f5dc7ac6c2736a4ebaf93fa in 2010). Since the introduction of Jetty as the embedded HTTP server (commit 4b77733e59dd8af742ff826c90c8d40b542b7563 in 2013), this is no more necessary as Jetty internals already do this for us in org.eclipse.jetty.util.UrlEncoded.decodeUtf8To(). So we can remove now this duplicated decoding as it prevents a proper use of the '+' character in search requests, as reported in issue #216. Changed Files: htroot/suggest.java, htroot/yacysearch.java |
Sat Aug 18 12:39:58 CEST 2018 by luccioman | Upgraded Apache PDFBox dependency from 2.0.9 to 2.0.11 Release notes at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310760&version=12343466 and https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310760&version=12342889 Changed Files: .classpath, build.xml, lib/fontbox-2.0.11.License, lib/fontbox-2.0.11.jar, lib/pdfbox-2.0.11.License, lib/pdfbox-2.0.11.jar, pom.xml |
Wed Aug 15 18:23:30 CEST 2018 by Erik Dominikus | Add contributor guidelines; closes #214 Changed Files: CONTRIBUTING.md |
Tue Aug 14 08:41:23 CEST 2018 by luccioman | Updated the JRE URL from 8u171 to 8u181 for the MS Windows installer Changed Files: build.nsi |
Sat Aug 11 10:02:26 CEST 2018 by luccioman | Fixed crawl queue folder naming for IPv6 hosts on MS Windows filesystems As reported by @vikulin in issue #187, crawling websites using a raw IPv6 address as host name in their URL failed when running on Microsoft Windows platforms (FAT32 or NTFS filesystems) when YaCy crawler created the crawl queue folder, as the ':' character which is part of an IPV6 address is forbidden on these filesystems. Changed Files: source/net/yacy/crawler/HostQueue.java |
Wed Aug 08 08:11:02 CEST 2018 by luccioman | Added a link to MediaWiki dumps summary in import page for convenience Changed Files: htroot/IndexImportMediawiki_p.html, locales/de.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Wed Aug 08 08:06:24 CEST 2018 by luccioman | Upgraded commons-compress dependency from version 1.16.1 to 1.17 Changed Files: .classpath, build.xml, lib/commons-compress-1.17.License, lib/commons-compress-1.17.jar, pom.xml |
Wed Aug 08 08:04:33 CEST 2018 by luccioman | Cleaned up unused old jar files not removed on previous Solr upgrade Changed Files: |
Mon Aug 06 12:07:47 CEST 2018 by luccioman | Prevent entering empty OpenSearch URLs in ConfigHeuristics_p.html In order to early prevent adding invalid configuration entries to the heuristicopensearch.conf file, as revealed the issue #209. Changed Files: htroot/ConfigHeuristics_p.html |
Fri Aug 03 18:39:41 CEST 2018 by luccioman | Upgraded the OpenGeoDB dump URL The status of the library in the DictionaryLoader_p.html page now also advertises the user that an upgrade can be applied when an older dump is already loaded. Upgrade applied as suggested by Niklas Andrus @fapth_gitlab on Gitter chat. Changed Files: htroot/DictionaryLoader_p.html, htroot/DictionaryLoader_p.java, source/net/yacy/document/LibraryProvider.java |
Tue Jul 31 16:07:08 CEST 2018 by luccioman | Adjusted yjson Solr writer to support responses from an external Solr Worked previously only with responses from YaCy embedded Solr, now able to render the response when YaCy is configured to use an external Solr index. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java |
Tue Jul 31 16:02:50 CEST 2018 by luccioman | Simplified a little bit the RSS OpenSearch Solr writer Changed Files: source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java |
Sun Jul 29 16:30:37 CEST 2018 by luccioman | Fixed a NPE case on the Solr OpenSearch response writer Occurred when omitHeader parameter is set to true Changed Files: source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java |
Sat Jul 28 11:03:31 CEST 2018 by luccioman | Made Solr OpenSearch RSS writer compatible with external Solr index Worked previously only with responses from YaCy embedded Solr, now able to render the response when YaCy is configured to use an external Solr index. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Thu Jul 26 09:39:54 CEST 2018 by luccioman | Upgraded icu4j dependency from 61.1 to 62.1 Changed Files: .classpath, build.xml, lib/icu4j-62.1.jar, lib/icu4j.License, pom.xml |
Thu Jul 26 08:22:13 CEST 2018 by luccioman | Merge pull request #188 from tangdou1/patch-3 small update in zh.lng Changed Files: locales/zh.lng |
Wed Jul 25 08:03:25 CEST 2018 by luccioman | Fixed stylesheet relative URLs rendering in Solr html writer Relative URLs to CSS stylesheets were not properly rendered when using the Solr html response writer and the "/solr/collection1/select" entry point instead of "/solr/select". Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Tue Jul 24 10:13:55 CEST 2018 by luccioman | Improved rendering of the Solr api relative url in the html writer In order to have a consistent relative url when using either /solr/select or /solr/collection1/select entry point. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Mon Jul 23 18:36:57 CEST 2018 by luccioman | Optionally render the response header when using the Solr html writer With params rendered as html input fields for conveniently modifying params values and refreshing results. Changed Files: htroot/ConfigSearchPage_p.html, htroot/IndexControlURLs_p.html, htroot/ViewFile.html, htroot/yacysearchitem.html, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java |
Mon Jul 23 13:50:24 CEST 2018 by luccioman | Override qf and df Solr params with defaults only when they are not set Changed Files: source/net/yacy/http/servlets/SolrSelectServlet.java |
Mon Jul 23 11:04:54 CEST 2018 by tangdou1 | small update in zh.lng Changed Files: locales/zh.lng |
Tue Jul 17 07:36:39 CEST 2018 by luccioman | Consistently render empty facets and facets having only entries at zero Changed Files: htroot/yacysearchtrailer.java |
Tue Jul 17 07:03:07 CEST 2018 by luccioman | Merge pull request #186 from tangdou1/patch-1 Update zh.lng Changed Files: locales/zh.lng |
Mon Jul 16 14:18:02 CEST 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Mon Jul 16 13:53:20 CEST 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Mon Jul 16 12:04:37 CEST 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Mon Jul 16 09:20:23 CEST 2018 by luccioman | Removed redundant synchronization lock on network switch function Was useless as done in an already synchronized block, and the lock object was assigned a new value in that same block, and nowhere else a lock is requested on that same object. Changed Files: source/net/yacy/search/Switchboard.java |
Mon Jul 16 09:13:50 CEST 2018 by luccioman | Removed unnecessary synchronization lock from serverSwitch constructor Lock was useless here as it was set on an object instance attribute while the object itself is not yet constructed and no other threads can access it. Changed Files: source/net/yacy/server/serverSwitch.java |
Sat Jul 14 15:36:03 CEST 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Sat Jul 14 11:18:43 CEST 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Fri Jul 13 07:25:58 CEST 2018 by luccioman | Use standard Java annotation syntax instead of custom Javadoc tag For better support by building tools. As reported by @KnustJohn_twitter , the custom [@phase](https://maven.apache.org/plugin-tools/maven-plugin-tools-java/index.html) Javadoc tag made NetBeans fail on Javadoc generation for the GitRevmavenTask class. Using instead standard Java 5 [annotations](https://maven.apache.org/plugin-tools/maven-plugin-plugin/examples/using-annotations.html#POM_configuration) this is no more an issue. Changed Files: libbuild/GitRevMavenTask/src/GitRevMavenTask.java |
Fri Jul 13 07:12:57 CEST 2018 by luccioman | Ignore Eclipse projects config files derived from maven pom.xml Changed Files: libbuild/GitRevMavenTask/.gitignore, libbuild/J7Zip-modified/.gitignore |
Wed Jul 11 08:13:29 CEST 2018 by luccioman | Fixed exceeding max size of failreason_s Solr field on large link list When using the 'From Link-List of URL' as a crawl start, with lists in the order of one or more thousands of links, the failreason_s Solr field maximum size (32kb) was exceeded by the string representation of the URL must-match filter when a crawl URL was rejected because not matching. Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/search/Switchboard.java |
Sun Jul 08 08:57:59 CEST 2018 by luccioman | Properly lock solrInstances for reboot and restoration of embedded Solr Putting a synchronization lock directly on the solrInstances property was ineffective as it is assigned a new (unlocked) instance in these operations. Changed Files: source/net/yacy/search/index/Fulltext.java |
Sun Jul 08 08:14:07 CEST 2018 by luccioman | Suppress compilation warning on unit testing intentional failure Changed Files: test/java/net/yacy/repository/BlacklistTest.java |
Thu Jul 05 09:36:36 CEST 2018 by luccioman | Fixed removal of URLs from the delegatedURL remote crawl stack URLs were removed from the stack using their hash as a bytes array, whereas the hash is stored in the stack as String instance. Changed Files: htroot/yacy/crawlReceipt.java, source/net/yacy/crawler/data/CrawlQueues.java |
Mon Jul 02 10:00:40 CEST 2018 by luccioman | Removed remaining unsafe accesses to SimpleDateFormat instances. SimpleDateFormat must not be used by concurrent threads without synchronization for parsing or formating dates as it is not thread-safe (internally holds a calendar instance that is not synchronized). Prefer now DateTimeFormatter when possible as it is thread-safe without concurrent access performance bottleneck (does not internally use synchronization locks). Changed Files: source/net/yacy/cora/federate/solr/responsewriter/GSAResponseWriter.java, source/net/yacy/data/wiki/WikiBoard.java, source/net/yacy/document/DateDetection.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/kelondro/blob/MapDataMining.java |
Mon Jul 02 00:00:04 CEST 2018 by reger | update bootstrap hosts Changed Files: defaults/yacy.network.freeworld.unit |
Sat Jun 30 14:27:37 CEST 2018 by luccioman | Updated the JRE URL from 8u131 to 8u171 for the MS Windows installer Changed Files: build.nsi |
Fri Jun 29 15:49:55 CEST 2018 by luccioman | Removed more unsafe concurrent accesses to SimpleDateFormat instances. SimpleDateFormat must not be used by concurrent threads without synchronization for parsing or formating dates as it is not thread-safe (internally holds a calendar instance that is not synchronized). Prefer now DateTimeFormatter when possible as it is thread-safe without concurrent access performance bottleneck (does not internally use synchronization locks). Changed Files: htroot/CrawlResults.java, htroot/IndexCreateQueues_p.java, source/net/yacy/cora/document/feed/RSSMessage.java, source/net/yacy/cora/protocol/HeaderFramework.java, test/java/net/yacy/cora/protocol/HeaderFrameworkTest.java |
Thu Jun 28 15:34:53 CEST 2018 by luccioman | Fixed Table_API_p.html current table page loss on row editing. Reset only to the first table page when the search query is modified Changed Files: htroot/Table_API_p.html |
Fri Jun 22 10:49:01 CEST 2018 by luccioman | Fixed a NullPointerException case in the suggest api Changed Files: htroot/suggest.java |
Tue Jun 19 12:53:17 CEST 2018 by luccioman | Made CrawlProfile loading tolerant to malformed json string attribute Changed Files: source/net/yacy/crawler/data/CrawlProfile.java |
Tue Jun 19 12:48:17 CEST 2018 by luccioman | Do not block whole server startup on persisted crawl profile load error Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java |
Tue Jun 19 12:46:50 CEST 2018 by luccioman | Properly render json string attributes in the crawl profile html editor Changed Files: htroot/CrawlProfileEditor_p.java |
Tue Jun 19 00:42:23 CEST 2018 by Michael Christen | removed transformer it was not used any more Changed Files: htroot/Crawler_p.java, source/net/yacy/crawler/FileCrawlStarterTask.java, source/net/yacy/data/BookmarkHelper.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java |
Tue Jun 12 14:19:22 CEST 2018 by luccioman | Added Italian in available web interface languages list Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java |
Tue Jun 12 14:15:29 CEST 2018 by luccioman | Merge pull request #183 from SebastianoPistore/master Added Italian translation. Changed Files: locales/it.lng |
Mon Jun 11 11:38:58 CEST 2018 by luccioman | Allow edtion of scheduled next execution dates for finer control Can be useful more especially when scheduling many API calls over a long period of time to precisely adjust each scheduled date/time. Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java, source/net/yacy/cora/date/GenericFormatter.java |
Sun Jun 10 15:11:50 CEST 2018 by Sebastiano Pistore | Added Italian translation. Changed Files: locales/it.lng |
Fri Jun 08 11:18:29 CEST 2018 by luccioman | Use the heavy ConcurrentUpdateSolrClient only when necessary Prefer the lightweight HttpSolrClient when no updates are performed on the remote Solr instance, as recommended by Solr documentation itself. Changed Files: source/net/yacy/cora/federate/SolrFederateSearchConnector.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/AutoSearch.java |
Fri Jun 08 10:33:23 CEST 2018 by luccioman | Add a max acceptable limit to the size of Solr responses on p2p search Following activation of gzip compression on responses, to ensure uncompressed content can fit on available memory. Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/StrictSizeLimitEntityWrapper.java, source/net/yacy/cora/protocol/http/StrictSizeLimitResponseInterceptor.java, source/net/yacy/peers/Protocol.java |
Thu Jun 07 09:40:54 CEST 2018 by luccioman | Consistently allow gzip compression of remote Solr responses Was already enabled when requesting remote Solr with https or with authentication (as an external Solr index) Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Wed Jun 06 14:24:05 CEST 2018 by luccioman | Reuse expired connections evictors threads provided by apache and solr Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/HTTPClient.java |
Wed Jun 06 09:36:50 CEST 2018 by luccioman | Made outgoing pools max total connections user configurable For a finer control over the maximum simultaneously active outgoing connections. Changed Files: defaults/yacy.init, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java |
Tue Jun 05 13:35:39 CEST 2018 by luccioman | Added gzip compression of responses returned to user-agents accepting it Enabled as default, but can be disabled using the "Server Access Settings" admin page. Changed Files: defaults/yacy.init, htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_ServerAccess.inc, htroot/Settings_p.java, source/net/yacy/http/Jetty9HttpServerImpl.java, source/net/yacy/search/SwitchboardConstants.java |
Sat Jun 02 17:38:14 CEST 2018 by luccioman | Apply remote solr configured timeout on getting connection from pool Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Tue May 29 09:24:21 CEST 2018 by luccioman | Use a common pooled http connection manager for remote solr instances For a better control on the maximum simultaneous outgoing http connections, as already done for any other http connections (crawls, rwi search, p2p protocol) using the net.yacy.cora.protocol.http.HTTPClient Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/search/Switchboard.java, source/net/yacy/yacy.java |
Sat May 26 21:00:24 CEST 2018 by luccioman | Removed use of deprecated ConcurrentUpdateSolrClient constructor Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Sat May 26 20:58:05 CEST 2018 by luccioman | Trace level log message for monitoring remote solr response times Changed Files: source/net/yacy/peers/Protocol.java |
Fri May 25 11:13:43 CEST 2018 by luccioman | Added a search page customization setting to display or not favicons If not interested in displaying this on your search results and notably on a peer with limited resources this can help saving some CPU and outgoing network connections. Changed Files: defaults/yacy.init, htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/yacysearchitem.java, source/net/yacy/search/SwitchboardConstants.java |
Thu May 24 09:24:52 CEST 2018 by luccioman | Added missing default Solr http client connection timeout initialization Consistently with the custom Solr http client used for https connections to remote Solr peers or to YaCy external Solr storage. This prevent remote Solr requests threads to wait for establishing a connection to a remote peer longer than the configured timeout. Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Tue May 22 07:55:09 CEST 2018 by luccioman | Limit the rate of calls to the suggest API when typing in search field Changed Files: htroot/index.html, htroot/yacysearch.html |
Mon May 21 14:49:43 CEST 2018 by luccioman | Adjusted suggestions timeout management * less CPU usage using the Solr 'allowedTime' parameter * increase chances to get some results even when a first operation step goes in time out by letting some time for final snippets results processing Changed Files: source/net/yacy/data/DidYouMean.java |
Sun May 20 21:51:41 CEST 2018 by reger | upd to slf4j-1.7.25 Changed Files: .classpath, build.xml, lib/jcl-over-slf4j-1.7.25.jar, lib/log4j-over-slf4j-1.7.25.jar, lib/slf4j-api-1.7.25.jar, lib/slf4j-jdk14-1.7.25.jar, pom.xml |
Fri May 18 10:14:54 CEST 2018 by luccioman | Properly handle embedded Solr partial results Solr can provide partial results for example when a processing time limit (specified with the parameter `timeAllowed`) is exceeded. Before this fix, getting partial results from an embedded Solr index resulted in a ClassCastException : "org.apache.solr.common.SolrDocumentList cannot be cast to org.apache.solr.response.ResultContext". Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Mon May 14 15:26:44 CEST 2018 by luccioman | Prevent duplication of Solr query highlight fields parameters That was caused by concurrent modifications (with addHighlightField() function) to the same SolrQuery instance when requesting Solr on remote peers in p2p search. Changed Files: source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/QueryParams.java |
Sun May 13 10:29:52 CEST 2018 by luccioman | Reduced memory footprint of text snippet extraction By not parsing and storing at first all sentences of a document, but only on the fly the ones necessary to compute the snippet. Changed Files: source/net/yacy/document/SentenceReader.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/search/snippet/TextSnippet.java, test/java/net/yacy/search/snippet/TextSnippetTest.java |
Fri May 11 15:42:53 CEST 2018 by luccioman | Reduced text snippet extraction processing time. By not generating MD5 hashes on all words of indexed texts, processing time is reduced by 30 to 50% on indexed documents with more than 1Mbytes of plain text. Changed Files: htroot/ViewFile.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/snippet/TextSnippet.java, test/java/net/yacy/search/snippet/TextSnippetTest.java |
Sun May 06 00:49:30 CEST 2018 by reger | upd to jwat-warc-1.1.1 Changed Files: .classpath, build.xml, lib/jwat-archive-common-1.1.1.jar, lib/jwat-common-1.1.1.jar, lib/jwat-gzip-1.1.1.jar, lib/jwat-warc-1.1.1.jar, pom.xml |
Thu May 03 09:00:09 CEST 2018 by luccioman | Removed unnecessary html end ligne tag with invalid syntax Changed Files: locales/ru.lng |
Thu May 03 08:53:07 CEST 2018 by luccioman | Upgraded ConfigNetwork_p html doctype and added language attribute Changed Files: htroot/ConfigNetwork_p.html, locales/de.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng, locales/zh.lng |
Thu May 03 08:37:38 CEST 2018 by luccioman | Removed unpaired select closing tag Changed Files: htroot/ConfigNetwork_p.html |
Wed May 02 08:38:58 CEST 2018 by luccioman | Added a UI section to configure encryption of peers communications Changed Files: htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java |
Tue May 01 14:04:15 CEST 2018 by sgaebel | corrected icons_sizes_sxt to SolrType.string Changed Files: source/net/yacy/search/schema/CollectionSchema.java |
Tue May 01 09:58:05 CEST 2018 by luccioman | Easier tracking of longest text snippets initializations When text snippets statistics are enabled and FINE log level is enabled on the TextSnippetStatistics class. Changed Files: source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/snippet/TextSnippetStatistics.java, test/java/net/yacy/search/snippet/TextSnippetTest.java |
Mon Apr 30 04:59:34 CEST 2018 by reger | Make sure for image resource url enabled index image pixel size fields are filled if at least one of the image size fields is enabled in index (images_height_val, images_width_val, images_pixel_val). Previously all fields were required to be enabled (hint: default setting is height + width enabled) Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java, source/net/yacy/search/schema/CollectionSchema.java |
Sun Apr 29 08:26:53 CEST 2018 by luccioman | Removed more remaining uses of deprecated Seed.getIP() function. Changed Files: htroot/CrawlStartScanner_p.java, source/net/yacy/peers/Network.java |
Sat Apr 28 23:24:24 CEST 2018 by reger | upd to jsoup-1.11.3 Changed Files: .classpath, build.xml, lib/jsoup-1.11.3.jar, pom.xml |
Tue Apr 24 08:08:24 CEST 2018 by luccioman | Support for preferred https in peers communication on more operations Changed Files: htroot/sharedBlacklist_p.java, source/net/yacy/search/AutoSearch.java |
Tue Apr 24 08:05:17 CEST 2018 by luccioman | Keep https in result message URL when push_p API is requested over https Changed Files: htroot/api/push_p.java |
Mon Apr 23 08:07:56 CEST 2018 by luccioman | Updated call to function deprecated since commons-codec version 1.11 Changed Files: source/net/yacy/data/TransactionManager.java |
Wed Apr 18 09:18:49 CEST 2018 by luccioman | Removed Docker Cloud deploy button as service will soon be shut down See Docker notification at http://success.docker.com/article/cloud-migration Changed Files: README.md |
Wed Apr 18 08:10:51 CEST 2018 by luccioman | Enforced access controls on basic administration settings pages. Ensuring http post method is used for operations with server-side effects (in respect of http semantics), and a valid transaction token is provided by the user-agent. Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java |
Tue Apr 17 08:16:37 CEST 2018 by luccioman | Added a start script option to run as a foreground process without JMX Contrary to the -d/--debug option which opens the 9999 port thus allowing remote monitoring with JVM tools such as JConsole. Changed Files: docker/Dockerfile, docker/Dockerfile.alpine, startYACY.sh |
Sat Apr 14 16:16:35 CEST 2018 by reger | upd to icu4j-61.1 Changed Files: .classpath, build.xml, lib/icu4j-61.1.jar, lib/icu4j.License, pom.xml |
Thu Apr 12 09:23:00 CEST 2018 by luccioman | Added Solr HTML writer support for responses from remote instances Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/SolrjResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Wed Apr 11 07:56:41 CEST 2018 by luccioman | Added controls on mode switch when attached to remote Solr instance(s) - to prevent unwanted exposure of index entries about private local/intranet documents when switching from "Intranet Indexing" mode while attached to remote Solr instance(s) - to warn user about remote Solr instance(s) still attached when switching from modes other than "Intranet Indexing" Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java |
Tue Apr 10 11:15:31 CEST 2018 by luccioman | Improve rendering of remote Solr admin URLs - properly handle IPv6 loopback address replacement - replace loopback address or host only when accessing peer remotely - replace loopback part with the peer hostname as requested rather than with its seed public IP as this works better for Intranet mode and when peer is behind a reverse proxy. Changed Files: htroot/IndexFederated_p.java, source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/federate/solr/instance/ShardInstance.java |
Mon Apr 09 00:01:45 CEST 2018 by luccioman | Removed unnecessary "/admin" suffix from remote Solr instance admin URL For quite quite a long time now, the Solr /admin URL suffix indeed redirects to the Solr base context (see https://issues.apache.org/jira/browse/SOLR-3337) Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Sun Apr 08 20:13:53 CEST 2018 by reger | upd to pdfbox-2.0.9 Changed Files: .classpath, build.xml, lib/fontbox-2.0.9.License, lib/fontbox-2.0.9.jar, lib/pdfbox-2.0.9.License, lib/pdfbox-2.0.9.jar, pom.xml |
Sat Apr 07 13:42:06 CEST 2018 by luccioman | Ensure an embedded Solr is available for Solr dump/restore operations Otherwise, these operations triggered NullPointerException when only an external Solr index is attached. Changed Files: htroot/IndexExport_p.html, htroot/IndexExport_p.java, source/net/yacy/search/index/Fulltext.java |
Fri Apr 06 20:34:54 CEST 2018 by luccioman | Ensure no remote Solr is attached before "Shut Down and Re-Start Solr" Otherwise once this operation is applied, the remote Solr(s) instances are deconnected and the embedded Solr is connected even if disabled by setting "core.service.fulltext". Also use constants for related default setting values. Changed Files: htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/Fulltext.java |
Tue Apr 03 23:19:05 CEST 2018 by Marc Nause | Removed seed URLs pointing to server low.audioattack.de since it will not be updated anymore. Changed Files: defaults/yacy.network.freeworld.unit, defaults/yacy.network.metager.unit |
Tue Apr 03 15:36:17 CEST 2018 by luccioman | Fixed NPE case in Solr select servlet on external Solr only setup Regression introduced with commit 0d7625ecfb3c65f71ca320da74a701eaeb45abe7 Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java |
Tue Apr 03 15:34:44 CEST 2018 by luccioman | Updated external Solr setup basic instructions Changed Files: defaults/yacy.init |
Mon Apr 02 19:48:11 CEST 2018 by luccioman | Added hint message inciting to check accounts settings on fresh install When unrestricted access from localhost is set and the accounts config page has not been visited at all. Changed Files: htroot/Status.html, htroot/Status.java, locales/fr.lng, locales/master.lng.xlf |
Mon Apr 02 15:16:10 CEST 2018 by luccioman | Added a shared Java constant for setting key server.servlets.called Changed Files: htroot/Status.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/search/SwitchboardConstants.java |
Fri Mar 30 11:12:48 CEST 2018 by luccioman | Adjusted api table query matching strategies When inlined (for example in the CrawlProfileEditor_p.html page) : search only on the comment, as the url is not visible On regular display : search on comment OR url, instead of comment AND url. Otherwise searching on comments terms is almost useless as these terms are not necessarily present in the url. Changed Files: htroot/Table_API_p.java |
Fri Mar 30 10:53:06 CEST 2018 by luccioman | Fixed start record on the last api table results page When the last results page size was lower than maximumRecords, results from the previous page where displayed again. Changed Files: htroot/Table_API_p.java |
Fri Mar 30 10:21:42 CEST 2018 by luccioman | Enable api table page navigation with search query Applied the same default results page size as when a type filter is defined for proper and consistend page navigation when combining type filter and search query. Changed Files: htroot/Table_API_p.html, htroot/Table_API_p.java |
Thu Mar 29 14:24:25 CEST 2018 by luccioman | Display the total number of matching items when filtering on table API Notably for a proper page navigation of the crawl scheduler table (CrawlProfileEditor_p.html page). Changed Files: htroot/Table_API_p.java |
Tue Mar 27 18:05:20 CEST 2018 by luccioman | Added hint to clarify results rendered dates and 'Sort by date' switch Changed Files: htroot/ConfigSearchPage_p.html, htroot/yacysearchitem.html, htroot/yacysearchtrailer.html |
Tue Mar 27 07:42:26 CEST 2018 by luccioman | Keep the selected view mode when typing a new URL in the ViewFile page Otherwise, when interested in viewing `Link List` for example, each time you typed a new URL, `Parsed Sentences` view mode was selected as default and you had to selected again the view mode you are insterested in. Changed Files: htroot/ViewFile.html, htroot/ViewFile.java |
Mon Mar 26 16:26:46 CEST 2018 by luccioman | Updated link to Solr Function Queries documentation page Changed Files: htroot/RankingSolr_p.html |
Mon Mar 26 16:24:27 CEST 2018 by luccioman | Removed hard-coded patch for Solr 5.0 on ranking boost function The current default boost function (`recip(ms(NOW,last_modified),3.16e-11,1,1)`) for the Date ranking profile is indeed working fine. What can trigger the error `unexpected docvalues type NUMERIC for field 'last_modified'` is the previous default boost function (quite old now) or any custom one using the Solr `ord` or `rord` functions on the last_modified field. Then the problem was that the migration code in the Switchboard supposed to detect the old date boost function was incorrect (one trailing right parenthesis in excess), so the deprecated function remained. This fixes issue #169. Changed Files: source/net/yacy/cora/federate/solr/Ranking.java, source/net/yacy/search/Switchboard.java |
Fri Mar 23 10:49:39 CET 2018 by luccioman | Fixed error 414 (URI Too Long) when manually selecting to many RSS items Switched form method to HTTP POST to prevent this. Changed Files: htroot/Load_RSS_p.html |
Fri Mar 23 10:28:19 CET 2018 by luccioman | Added a crawl filtering possibility on documents Media Type (MIME) Changed Files: htroot/CrawlProfileEditor_p.xml, htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/Crawler_p.java, source/net/yacy/crawler/data/CrawlProfile.java, source/net/yacy/document/Document.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/ErrorCache.java |
Fri Mar 23 10:23:54 CET 2018 by luccioman | Updated link URL to IANA Media Types with https Changed Files: htroot/ConfigParser_p.html |
Wed Mar 21 08:22:29 CET 2018 by luccioman | Added support for enclosures (media links) to the RSS loader Changed Files: htroot/Load_RSS_p.html, htroot/Load_RSS_p.java, source/net/yacy/crawler/retrieval/RSSLoader.java |
Wed Mar 21 07:37:53 CET 2018 by luccioman | Updated Docker readme base image links from java (deprecated) to openjdk Changed Files: docker/Readme.md |
Tue Mar 20 10:09:17 CET 2018 by luccioman | Reduce log verbosity of RSS loader on feed items with no link Changed Files: htroot/Load_RSS_p.html, htroot/Load_RSS_p.java, source/net/yacy/crawler/retrieval/RSSLoader.java |
Tue Mar 20 07:38:29 CET 2018 by luccioman | Added RSS reader support for `enclosure` feed item sub element. Enclosure element (see http://www.rssboard.org/rss-specification#ltenclosuregtSubelementOfLtitemgt ) can be seen for example in podcasts feeds. Changed Files: source/net/yacy/cora/document/feed/Hit.java, source/net/yacy/cora/document/feed/RSSMessage.java, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/document/parser/rssParser.java |
Mon Mar 19 10:49:35 CET 2018 by luccioman | Added some JavaDoc to the RSSMessage class. Changed Files: source/net/yacy/cora/document/feed/RSSMessage.java |
Mon Mar 19 02:02:35 CET 2018 by reger | upd to commons-codec-1.11 Changed Files: .classpath, build.xml, lib/commons-codec-1.11.License, lib/commons-codec-1.11.jar, pom.xml |
Fri Mar 16 11:35:42 CET 2018 by luccioman | Handle Solr fields restrict and alias in YaCy html and exml writers Thus allowing for example to read more easily the local Solr index full metadata in HTML by restricting if desired to some fields of interest. See Solr documentation about the 'fl' (Field List) parameter at https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html#CommonQueryParameters-Thefl_FieldList_Parameter Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Thu Mar 15 09:59:57 CET 2018 by luccioman | Parse and index more common audio metadata text tag fields. Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Tue Mar 13 23:08:52 CET 2018 by luccioman | Added parsing of URLs eventually present in audio metadata tags Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Sat Mar 10 15:46:53 CET 2018 by Michael Peter Christen | added nav filter Changed Files: htroot/AccessTracker_p.java, htroot/CrawlStartExpert.html, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/document/parser/html/ContentScraper.java |
Thu Mar 01 20:50:44 CET 2018 by luccioman | Enabled partial parsing of audio resources. Changed Files: source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/kelondro/util/FileUtils.java |
Wed Feb 28 13:46:40 CET 2018 by luccioman | Updated audio file extensions with ones recently added to audioTagParser Changed Files: source/net/yacy/cora/document/analysis/Classification.java |
Wed Feb 28 12:27:17 CET 2018 by luccioman | Let a chance for other parsers on audioTagParser error As done in all other parsers, eventually falling back in the end to the genericParser which creates a minimal index entry. Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Wed Feb 28 11:58:32 CET 2018 by luccioman | Reuse existing File copy function to handle audio parser tmp files Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Wed Feb 28 08:19:13 CET 2018 by luccioman | Factored audio parser tag processing Changed Files: source/net/yacy/document/parser/audioTagParser.java |
Wed Feb 28 07:49:40 CET 2018 by luccioman | Removed some unnecessary intermediate list creation on array copy. Changed Files: source/net/yacy/document/Document.java |
Tue Feb 27 18:04:12 CET 2018 by luccioman | Updated the list of audio file formats supported by the audioTagParser Follows upgrade to Jaudiotagger dependency to version 2.2.5. Changed Files: defaults/yacy.init, source/net/yacy/document/parser/audioTagParser.java, source/net/yacy/migration.java, source/net/yacy/search/Switchboard.java |
Mon Feb 26 09:17:26 CET 2018 by luccioman | Upgraded Jaudiotagger dependency from 2.0.3 to 2.2.5 Changed Files: .classpath, build.xml, lib/jaudiotagger-2.2.5.License, lib/jaudiotagger-2.2.5.jar, pom.xml |
Fri Feb 23 19:17:09 CET 2018 by reger | upd to commons-compress-1.16.1 Changed Files: .classpath, build.xml, lib/commons-compress-1.16.1.jar, lib/commons-compress-1.16.License, pom.xml |
Fri Feb 23 11:41:50 CET 2018 by luccioman | Added HTML5 embedded audio for results playing on supporting browsers Restricted to authenticated or localhost users only to prevent redistribution license issues. Changed Files: htroot/env/base.css, htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Fri Feb 23 11:36:03 CET 2018 by luccioman | Added missing vocabulary navigator increment on results from RWI Changed Files: source/net/yacy/search/query/SearchEvent.java |
Wed Feb 21 08:41:13 CET 2018 by luccioman | Allow creation of vocabularies from remote CSV file URLs. Changed Files: htroot/Vocabulary_p.html, htroot/Vocabulary_p.java, source/net/yacy/kelondro/util/FileUtils.java |
Wed Feb 21 08:38:35 CET 2018 by luccioman | Make StreamResponse usable in Java try-with-resources statements Changed Files: source/net/yacy/crawler/retrieval/StreamResponse.java |
Tue Feb 20 12:22:54 CET 2018 by luccioman | Enforced controls on vocabulary editing operations. Changed Files: htroot/Vocabulary_p.html, htroot/Vocabulary_p.java |
Tue Feb 20 11:22:34 CET 2018 by luccioman | Vocabulary editor : use accessible labels and CSS for elements position Changed Files: htroot/Vocabulary_p.html, htroot/env/base.css |
Mon Feb 19 15:15:02 CET 2018 by luccioman | Vocabulary_p.html : richer semantics for HTML tables Also replaced deprecated attributes Changed Files: htroot/Vocabulary_p.html |
Mon Feb 19 11:48:40 CET 2018 by luccioman | Provide user interface messages on vocabulary creation read/write errors Changed Files: htroot/Vocabulary_p.html, htroot/Vocabulary_p.java, source/net/yacy/cora/lod/vocabulary/Tagging.java |
Mon Feb 19 09:35:44 CET 2018 by luccioman | Mark vocabulary name field as required using html instead of JavaScript Changed Files: htroot/Vocabulary_p.html |
Mon Feb 19 08:54:42 CET 2018 by luccioman | Fixed Vocabulary_p.html HTML validation errors. Validated with Validated with Nu Html Checker 17.11.1. Changed Files: htroot/Vocabulary_p.html, locales/de.lng, locales/master.lng.xlf, locales/ru.lng |
Fri Feb 16 10:19:41 CET 2018 by luccioman | Issue #156 : new option to clean up (or not) search cache on crawl start Prevent also unnecessary search event cache clean-up on each access to the crawl monitor page (Crawler_p.html). Changed Files: htroot/CrawlStartExpert.html, htroot/CrawlStartExpert.java, htroot/CrawlStartSite.html, htroot/Crawler_p.java, htroot/Load_MediawikiWiki.html, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.html, htroot/Load_PHPBB3.java |
Fri Feb 16 08:51:26 CET 2018 by luccioman | Upgraded maven JUnit test dependency from 4.11 to 4.12 Changed Files: pom.xml |
Thu Feb 15 19:14:07 CET 2018 by luccioman | Use https rather than http in links and queries to openstreetmap.org Changed Files: htroot/yacysearch.html, source/net/yacy/peers/graphics/OSMTile.java |
Thu Feb 15 07:29:17 CET 2018 by luccioman | Handle escaped line breaks and separators in vocabulary import from CSV Changed Files: htroot/Vocabulary_p.java, test/Vocabulary_pTest.java |
Wed Feb 14 10:31:09 CET 2018 by luccioman | Added a line start field for vocabulary import from CSV file As a convenience to ignore eventual CSV header lines Changed Files: htroot/Vocabulary_p.html, htroot/Vocabulary_p.java |
Wed Feb 14 09:29:04 CET 2018 by luccioman | Added option to choose field delimiter in vocabulary import from CSV Changed Files: htroot/Vocabulary_p.html, htroot/Vocabulary_p.java |
Wed Feb 14 09:27:17 CET 2018 by luccioman | Removed unused import Changed Files: source/net/yacy/search/query/SearchEvent.java |
Wed Feb 14 07:14:25 CET 2018 by luccioman | Reuse the same Pattern instance when matching multiple key/values Changed Files: source/net/yacy/server/serverObjects.java |
Tue Feb 13 18:24:26 CET 2018 by luccioman | Improved blacklist entries editing operations : - Fixes issue #160 : handle properly syntax exceptions with a user friendly message - Fixes loss of information on multiple blacklist entries editions - Fixes loss of entries when moving entries from one list to another Changed Files: htroot/Blacklist_p.html, htroot/Blacklist_p.java, htroot/IndexControlRWIs_p.java, htroot/api/blacklists/add_entry_p.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/BlacklistHelper.java, source/net/yacy/server/serverObjects.java |
Mon Feb 12 01:16:14 CET 2018 by reger | Remove now obsolete html for language-nav and ISO639 jar reference Changed Files: htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java |
Mon Feb 12 00:16:34 CET 2018 by reger | Adjust and move Language Navigator to be member of the navigatior plugin list. Changed Files: htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java, htroot/yacysearchtrailer.java, source/net/yacy/search/navigator/LanguageNavigator.java, source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/query/SearchEvent.java |
Sat Feb 10 20:01:35 CET 2018 by reger | upd to httpclient-4.5.5 Changed Files: .classpath, build.xml, lib/httpclient-4.5.5.jar, lib/httpcore-4.4.9.License, lib/httpcore-4.4.9.jar, lib/httpmime-4.5.5.jar, pom.xml |
Thu Feb 08 08:07:30 CET 2018 by luccioman | Fixed loss of "meanCount" search param when using facets or page buttons Then on new search queries, no suggestions at all could be displayed. Changed Files: htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/search/query/QueryParams.java |
Wed Feb 07 15:54:46 CET 2018 by luccioman | Do not clear all search modifiers when unselecting one modifier. Previously, when clicking a selected facet in the search results page to unselect it, all other eventually selected modifiers/facets were also removed. Changed Files: htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryParams.java |
Tue Feb 06 15:14:14 CET 2018 by luccioman | Remove old query terms from search results suggestions links. Especially when old terms were misspelled, suggestions links then provided most of the time empty results. Changed Files: htroot/yacysearch.java, source/net/yacy/search/query/QueryParams.java |
Tue Feb 06 12:33:44 CET 2018 by luccioman | Enable results suggestions (Did you Mean) even when RWI is not enabled. RWI is no more necessary for suggestions processing since commit c40ba51ca6effd97b4e8835b2deade0ff071ac96. Revealed by a question about spell check from ouahpiti on YaCy forum (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=6084 ). Changed Files: htroot/yacysearch.java |
Fri Feb 02 10:27:36 CET 2018 by luccioman | Refactoring : documented and extracted autotagging processing functions. Changed Files: source/net/yacy/document/Tokenizer.java, test/java/net/yacy/document/TokenizerTest.java |
Fri Feb 02 09:31:40 CET 2018 by luccioman | Added HTML microdata typed items parsing capability. This adds the possibility for the HTML parser to gather typed items URLs annotated in HTML tags with itemscope and itemtype attributes (see microdata specification https://www.w3.org/TR/microdata/ ), notably Types from the schema.org vocabulary, but also Types/Classes from any other vocabulary, such as the common ones listed in the RDFa core context ( https://www.w3.org/2011/rdfa-context/rdfa-1.1.html ). Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/Scraper.java, source/net/yacy/document/parser/html/TransformerWriter.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java |
Tue Jan 30 21:00:18 CET 2018 by luccioman | Create recrawl requests with the relevant crawl profile. Recrawl default profile was previously effectively used for crawl stacker acceptance check, but request entries were indeed still created with the "snippetGlobalText" profile. Changed Files: source/net/yacy/crawler/RecrawlBusyThread.java |
Mon Jan 29 18:34:47 CET 2018 by luccioman | Added an utility to generate/update XLIFF master file from lng files. Changed Files: htroot/Translator_p.java, source/net/yacy/utils/translation/GenerateMasterXliff.java, source/net/yacy/utils/translation/TranslationManager.java |
Mon Jan 29 16:51:00 CET 2018 by luccioman | Updated master and French translation for the IndexReIndexMonitor_p page Changed Files: htroot/IndexReIndexMonitor_p.html, locales/de.lng, locales/fr.lng, locales/master.lng.xlf |
Mon Jan 29 14:03:01 CET 2018 by luccioman | Moved dbtest to the test source folder. Changed Files: test/java/net/yacy/dbtest.java |
Mon Jan 29 14:00:43 CET 2018 by luccioman | Fixed NullPointerException case on Table init with relative file path. Can occur for example when running dbtest with relative test table file name (wihout explicit parent folder). Changed Files: source/net/yacy/kelondro/table/Table.java |
Mon Jan 29 13:56:37 CET 2018 by luccioman | Shutdown daemon threads at the end of dbtest Changed Files: source/net/yacy/dbtest.java |
Mon Jan 29 13:38:25 CET 2018 by luccioman | Replaced improper ByteBuffer.equals() implementation by Arrays.equals() Renamed also ByteBuffer.equals() to startsWith() as this is the appropriate function implementation semantics. Changed Files: htroot/IndexControlRWIs_p.java, htroot/Wiki.java, htroot/yacy/search.java, source/net/yacy/cora/util/ByteBuffer.java, source/net/yacy/dbtest.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/ranking/ReferenceOrder.java |
Sun Jan 28 12:41:56 CET 2018 by luccioman | Added a manual performance test for the HostBalancer. Consequently to the report in mantis 776 (http://mantis.tokeek.de/view.php?id=776). Running the perfs test with different control parameters seems to reveal that the YaCy's RowHandleMap used in the balancer depthCache is finally more efficient than for example the ConcurrentHashMap from JDK 8. Changed Files: test/java/net/yacy/crawler/HostBalancerTest.java |
Sat Jan 27 18:32:45 CET 2018 by reger | upd to metadata-extractor-2.11.0.jar Changed Files: .classpath, build.xml, lib/metadata-extractor-2.11.0.License, lib/metadata-extractor-2.11.0.jar, pom.xml |
Fri Jan 26 17:15:27 CET 2018 by luccioman | Removed time condition on HostBalancer initialization in JUnit test. Its initialization in main application usage remains asynchronous. Changed Files: source/net/yacy/crawler/HostBalancer.java, test/java/net/yacy/crawler/HostBalancerTest.java |
Fri Jan 26 10:31:13 CET 2018 by luccioman | Commit Solr index before simulating or starting recrawl job. This ensures up-to-date simulation query results, and recrawl processing. Changed Files: htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Fri Jan 26 09:50:40 CET 2018 by luccioman | Merge pull request #155 from JeremyRand/readme-typo-fixes Fix some typos in the README. Changed Files: README.md |
Fri Jan 26 05:34:31 CET 2018 by JeremyRand | Fix some typos in the README. Changed Files: README.md |
Thu Jan 25 07:57:56 CET 2018 by luccioman | Revised the RDFaParser main launcher for minimal proper operation. This parser is still not enabled in the main text parsers list. More would have to be done to make it functional. Changed Files: source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaTripleImpl.java |
Sat Jan 20 18:54:08 CET 2018 by luccioman | Fixed stored URL in web cache when redirection(s) occurs. Associate cached content to the last redirection location, instead of the first URL of a redirection(s) chain : - for proper base URL processing in parsers (fixes mantis 636 - http://mantis.tokeek.de/view.php?id=636) - to prevent duplicated content in Solr index when recrawling a redirected URL Changed Files: source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/repository/LoaderDispatcher.java |
Fri Jan 19 11:58:52 CET 2018 by luccioman | Automatically refresh running recrawl report when JavaScript is enabled. For users who would prefer to keep JavaScript disabled, a manual Refresh button is still available. Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, htroot/IndexReIndexMonitor_p.json, htroot/js/IndexReIndexMonitor.js, htroot/jslicense.html |
Fri Jan 19 10:18:35 CET 2018 by luccioman | Merge pull request #154 from tangdou1/master update chinese translation Changed Files: locales/zh.lng |
Tue Jan 16 10:16:14 CET 2018 by tangdou1 | Merge pull request #1 from tangdou1/tangdou1-patch-1 Update zh.lng Changed Files: locales/zh.lng |
Tue Jan 16 10:11:07 CET 2018 by tangdou1 | Update zh.lng translate some untranslated words to chinese. Changed Files: locales/zh.lng |
Tue Jan 16 08:35:54 CET 2018 by tangdou1 | Update zh.lng Changed Files: locales/zh.lng |
Mon Jan 15 18:32:34 CET 2018 by luccioman | Set reindex page to html5 and removed presentational only html tables. Changed Files: htroot/IndexReIndexMonitor_p.html |
Mon Jan 15 17:16:54 CET 2018 by luccioman | Removed unused duplicated HTML id on header hidden field Changed Files: htroot/env/templates/header.template |
Mon Jan 15 10:05:49 CET 2018 by luccioman | Removed unncessary reflection usage for workflow tasks. This improves code readability and maintainability (calls hierarchy are easier to read) and eventually performance. Changed Files: source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/kelondro/workflow/WorkflowTask.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/search/Switchboard.java |
Mon Jan 15 08:30:37 CET 2018 by luccioman | Added new recrawl job profile to the list of default crawl profiles Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java |
Mon Jan 15 08:06:28 CET 2018 by luccioman | Refresh recrawl job profile threshold date like other default profiles Changed Files: source/net/yacy/search/Switchboard.java |
Sat Jan 13 15:46:04 CET 2018 by luccioman | Added a specific default crawl profile for the recrawl job. - with only light constraint on known indexed documents load date, as it can already been controlled by the selection query, and the goal of the job is indeed to recrawl selected documents now - using the iffresh cache strategy Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Sat Jan 13 12:13:04 CET 2018 by luccioman | Added comments about crawl profiles recrawl cycles Changed Files: source/net/yacy/crawler/CrawlSwitchboard.java |
Sat Jan 13 12:07:56 CET 2018 by luccioman | More comprehensive log on rejected recrawls caused by date constraint Changed Files: source/net/yacy/crawler/CrawlStacker.java |
Fri Jan 12 11:47:13 CET 2018 by luccioman | Added more details to the recrawl job report Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Fri Jan 12 10:23:26 CET 2018 by luccioman | Add a query link to local Solr to browse selected recrawl candidates Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java |
Thu Jan 11 09:53:27 CET 2018 by luccioman | Display recrawl job report also when job is actively running Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java |
Wed Jan 10 17:05:53 CET 2018 by luccioman | Record recrawl calls to make them schedulable Changed Files: htroot/IndexImportMediawiki_p.java, htroot/IndexReIndexMonitor_p.java, source/net/yacy/data/WorkTables.java |
Tue Jan 09 22:33:15 CET 2018 by luccioman | Added a report info box about eventual last terminated recrawl job For easier monitoring of recrawls. Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Tue Jan 09 10:22:26 CET 2018 by luccioman | Added a stop condition to the Recrawl busy thread Changed Files: htroot/IndexReIndexMonitor_p.java, source/net/yacy/crawler/RecrawlBusyThread.java |
Mon Jan 08 21:20:46 CET 2018 by luccioman | Made possible to customize selection query before launching a recrawl Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, locales/de.lng, locales/master.lng.xlf, source/net/yacy/crawler/RecrawlBusyThread.java |
Sun Jan 07 15:25:16 CET 2018 by luccioman | Enforced controls (HTTP method, token) on ReIndex and ReCrawl operations Changed Files: htroot/IndexReIndexMonitor_p.html, htroot/IndexReIndexMonitor_p.java, htroot/IndexSchema_p.html, htroot/IndexSchema_p.java |
Tue Jan 02 10:21:07 CET 2018 by luccioman | Fixed SegmentTest test case time dependant occasional failures As highlighted by latest automated Travis builds. Changed Files: source/net/yacy/kelondro/rwi/IndexCell.java, test/java/net/yacy/search/index/SegmentTest.java |
Tue Jan 02 08:13:14 CET 2018 by luccioman | Added UI switch to control content domain constraint per search request Changed Files: htroot/ConfigPortal_p.html, htroot/index.html, htroot/index.java, htroot/yacysearch.html, htroot/yacysearch.java, htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, source/net/yacy/search/query/QueryParams.java |
Fri Dec 29 11:32:42 CET 2017 by luccioman | Added UI setting for strictness of content-type checking on media search Changed Files: htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java |
Thu Dec 28 03:13:42 CET 2017 by reger | upd to commons-io-2.6 Changed Files: .classpath, build.xml, lib/commons-io-2.6.License, lib/commons-io-2.6.jar, pom.xml |
Thu Dec 28 02:51:52 CET 2017 by reger | Make TokenizedStringNavigator (used for keyword search facet) active check case insensitive. As keywords are compared lower case, make sure user input keyword:Key or keyword:key will be shown as active in facet entry key. Changed Files: source/net/yacy/search/navigator/TokenizedStringNavigator.java |
Sun Dec 24 01:34:23 CET 2017 by reger | upd to httpclient-4.5.4 and httpmime-4.5.4 Changed Files: .classpath, build.xml, lib/httpclient-4.5.4.jar, lib/httpmime-4.5.4.jar, pom.xml |
Sun Dec 24 01:02:18 CET 2017 by reger | upd to icu4j-60.2 Changed Files: .classpath, build.xml, lib/icu4j-60.2.License, lib/icu4j-60.2.jar, pom.xml |
Fri Dec 22 11:39:30 CET 2017 by luccioman | Enable full size images preview for users with extended search rights Changed Files: source/net/yacy/visualization/ImageViewer.java |
Fri Dec 22 11:01:02 CET 2017 by luccioman | Added UI setting for optional encryption with https on p2p searches Changed Files: htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java |
Thu Dec 21 18:41:32 CET 2017 by luccioman | Added optional https support for remote crawl and profile operations Changed Files: htroot/ViewProfile.java, htroot/rct_p.java, source/net/yacy/crawler/data/CrawlQueues.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java |
Tue Dec 19 12:30:49 CET 2017 by luccioman | Enable optional https support for /yacy/transferURL API calls. Also updated some Javadoc and consistently use Switchboard instance as a constructor parameter where relevant. Changed Files: htroot/IndexControlRWIs_p.java, source/net/yacy/peers/Dispatcher.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/Transmission.java, source/net/yacy/search/Switchboard.java |
Tue Dec 19 11:14:20 CET 2017 by luccioman | Updated links to Java Regular Expressions documentation to version 8 Changed Files: htroot/Blacklist_p.html, htroot/CrawlStartExpert.html, htroot/RegexTest.html, locales/hi.lng, locales/uk.lng, locales/zh.lng |
Sat Dec 16 00:49:48 CET 2017 by reger | upd to commons-compress-1.15 Changed Files: .classpath, build.xml, lib/commons-compress-1.15.License, lib/commons-compress-1.15.jar, pom.xml |
Fri Dec 15 17:03:35 CET 2017 by luccioman | Restored peer URL host name stripping removed from previous commit. Still useful for peers with IPv6 addresses. Changed Files: source/net/yacy/peers/Protocol.java |
Wed Dec 13 07:38:04 CET 2017 by luccioman | Merge pull request #149 from Scre13/bugfix_default_settings Fixed loading default thread load setting in Performance Settings of Queues and Processes. Changed Files: htroot/PerformanceQueues_p.java |
Tue Dec 12 23:25:56 CET 2017 by ScRe13 | fixed default loading default settings; load was populated with wrong value Changed Files: htroot/PerformanceQueues_p.java |
Sun Dec 10 01:25:20 CET 2017 by reger | Show hide or show public surftip button depending on current config status, to show the button to switch the status (hiding button of current status) Changed Files: htroot/Surftips.html, htroot/Surftips.java |
Fri Dec 08 15:26:46 CET 2017 by luccioman | Removed Java 1.8 no more necessary version checking (fixes issue #147) Java 1.8 is by the way now a prerequisite to run from latest sources. Changed Files: htroot/Status.html, htroot/Status.java |
Fri Dec 08 01:01:07 CET 2017 by reger | remove deprecated jetty continuation class from urlproxyservlet (was a long time carry over, while not supporting async requests) Changed Files: source/net/yacy/http/servlets/UrlProxyServlet.java |
Thu Dec 07 15:16:11 CET 2017 by Michael Peter Christen | (more!) evaluation of XRealIP from nginx reverse proxy Changed Files: htroot/yacysearchitem.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/http/Jetty9YaCySecurityHandler.java, source/net/yacy/http/MonitorHandler.java, source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/UrlProxyServlet.java, source/net/yacy/http/servlets/YaCyQoSFilter.java |
Mon Dec 04 19:13:16 CET 2017 by luccioman | Made "tld:" modifier case insensitive and IDN complient. Thus allowing typing internationalized top-level domains with non ASCII characters as tld: modifier. Changed Files: htroot/yacysearch.java |
Mon Dec 04 18:23:26 CET 2017 by luccioman | Improved support for internationalized domain names on "site:" modifier Allow typing directly internationalized domain names including non ASCII characters in the search field. Search is done using the ASCII Compatible Encoding (ACE) representation. Changed Files: source/net/yacy/search/query/QueryModifier.java |
Mon Dec 04 14:11:29 CET 2017 by luccioman | Do locale independant case conversion on "filetype:" query modifier. Changed Files: source/net/yacy/search/query/QueryModifier.java |
Mon Dec 04 14:08:34 CET 2017 by luccioman | Made "site:" query modifier case insensitive. Changed Files: source/net/yacy/search/query/QueryModifier.java |
Mon Dec 04 13:58:15 CET 2017 by luccioman | Refactored 'site:' query modifier parsing into a dedicated function. Changed Files: source/net/yacy/search/query/QueryModifier.java |
Mon Dec 04 01:12:50 CET 2017 by reger | upd to httpcore-4.4.8 Changed Files: .classpath, build.xml, lib/httpcore-4.4.8.License, lib/httpcore-4.4.8.jar, pom.xml |
Sat Dec 02 08:45:42 CET 2017 by luccioman | Merge pull request #144 from him2him2/_fic_HTTPS Update HTTP -> HTTPS in README.md Changed Files: README.md |
Fri Dec 01 11:52:52 CET 2017 by luccioman | Prefer fine URL match over approximate URL mask regex on final filtering Also prevent adding a redundant and CPU costly Solr url mask filter query when possible Changed Files: source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java |
Fri Dec 01 11:19:31 CET 2017 by luccioman | Improved accuracy of URLs search filters : protocol, tld, host, file ext Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java, test/java/net/yacy/search/query/QueryParamsTest.java |
Fri Dec 01 08:46:46 CET 2017 by luccioman | Apply tld query modifier on Solr host_s mandatory field. The filter has thus much more chances to be effective than when applied on the optional field host_dnc_s. Changed Files: source/net/yacy/search/query/QueryParams.java |
Thu Nov 30 09:20:32 CET 2017 by luccioman | Refactored url mask filter build from query modifiers For better readability and easier unit testing. Changed Files: source/net/yacy/search/query/QueryParams.java |
Sun Nov 26 22:01:42 CET 2017 by reger | upd to Jsoup-1.11.2 Changed Files: .classpath, build.xml, lib/jsoup-1.11.2.jar, pom.xml |
Sun Nov 26 02:53:51 CET 2017 by reger | remove redundant setting of timeout for remoteinstance and replace depreciated updatesolrclient instantiation with recommended builder Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java |
Thu Nov 23 09:54:36 CET 2017 by Ronald Eddy Jr | Update HTTP -> HTTPS in README.md URLs were updated to use HTTPS protocol in README.md. Changed Files: README.md |
Wed Nov 22 09:07:36 CET 2017 by luccioman | Upgraded apache POI dependency from 3.16 to 3.17 Changed Files: .classpath, build.xml, lib/poi-3.17.License, lib/poi-3.17.jar, lib/poi-scratchpad-3.17.jar, pom.xml |
Wed Nov 22 09:06:16 CET 2017 by luccioman | Added a basic JUnit test for the Visio parser (vsdParser) Changed Files: source/net/yacy/document/parser/vsdParser.java, test/java/net/yacy/document/parser/vsdParserTest.java |
Mon Nov 20 18:52:45 CET 2017 by luccioman | Do locale neutral case conversion of HTML charset name. Required to properly run on systems with default locale set to Turkish language, as with this locale the 'i' character has different upper and lower case flavors than with other locales. Changed Files: source/net/yacy/document/parser/htmlParser.java |
Mon Nov 20 18:50:49 CET 2017 by luccioman | Restore initial locale at the end of a JUnit test case which modify it. Changed Files: test/java/net/yacy/document/TextParserTest.java |
Mon Nov 20 18:47:46 CET 2017 by luccioman | Do locale neutral case conversions on domain names. Required to properly run on systems with default locale set to Turkish language, as with this locale the 'i' character has different upper and lower case flavors than with other locales. Changed Files: source/net/yacy/cora/protocol/Domains.java, test/java/net/yacy/cora/protocol/DomainsTest.java |
Mon Nov 20 15:23:33 CET 2017 by luccioman | Do locale neutral case conversions in MultiProtocolURL For any relevant URL parts : host name, URL scheme, session ids or technical parts (see https://url.spec.whatwg.org/#url-writing and https://tools.ietf.org/html/rfc3986 for current standard references). Remaining locale sensitive conversion used for detection of URL word components in urlComps() makes sense but using detected language would be preferable than using the default system locale. Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Mon Nov 20 09:48:46 CET 2017 by luccioman | Do locale neutral case conversions in Classification Required for people using Turkish language as their default system locale, as with this locale the 'i' character has different upper and lower case flavors than with other locales. Changed Files: source/net/yacy/cora/document/analysis/Classification.java, test/java/net/yacy/cora/document/analysis/ClassificationTest.java |
Fri Nov 17 11:09:55 CET 2017 by luccioman | Added signing key to developer releases location. Changed Files: defaults/yacy.network.freeworld.unit |
Thu Nov 16 09:50:55 CET 2017 by luccioman | Updated lists of known sponsored and country-code TLDs. Using current IANA reference list at https://www.iana.org/domains/root/db . As for previous update on known generic TLDs list, the generated URL hashes on these domains stay the same but it improves performance of URL hash computation for URLs on these domains. Changed Files: source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/tld/GenericTLD.java, source/net/yacy/cora/protocol/tld/InternationalizedCountryCodeTLD.java, source/net/yacy/cora/protocol/tld/SponsoredTLD.java |
Tue Nov 14 09:42:09 CET 2017 by luccioman | Updated the generic top-level known domains list. Using current IANA reference list at https://www.iana.org/domains/root/db The generated URL hashes on these domains stay the same but performance is greatly improved as a DNS resolve request is required on URL hash computation when the TLD part of the host name is unknown. Hash computation mean time measured on 1541 sample URLs (one on each TLD) and a computer with a DSL connection : about 230ms before change, then only 20ms. Changed Files: source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/GenericTLD.java, test/java/net/yacy/cora/document/id/DigestURLHashPerfTest.java |
Tue Nov 14 09:24:13 CET 2017 by luccioman | Added some JavaDoc Changed Files: source/net/yacy/kelondro/util/FileUtils.java |
Tue Nov 14 09:17:43 CET 2017 by luccioman | Updated log path in informative message of stop script. As highlighted by @Lew-Rockwell-Fan in issue #140, the two log paths mentioned by the stopYACY.sh script were inconsistent. Changed Files: stopYACY.sh |
Wed Nov 08 09:33:30 CET 2017 by luccioman | Improved some JUnit tests isolation and resources release The modified tests were successfull when run manually from an IDE such as Eclipse, but failed occasionnally when run with maven as part of the overall test suite. Changed Files: test/java/net/yacy/kelondro/io/RecordsTest.java, test/java/net/yacy/search/index/SegmentTest.java |
Tue Nov 07 19:02:09 CET 2017 by luccioman | Remove old hard-coded holiday dates from DateDection class. Replaced with rules based relative to current year as already done for a part of the supported dates. Changed Files: source/net/yacy/document/DateDetection.java, test/java/net/yacy/document/DateDetectionTest.java |
Mon Nov 06 09:37:44 CET 2017 by luccioman | Upgraded icu4j dependency from 59_1 to 60.1 Changed Files: .classpath, build.xml, lib/icu4j-60.1.License, lib/icu4j-60.1.jar, pom.xml |
Mon Nov 06 09:14:03 CET 2017 by luccioman | Added a html parser charset detection unit test Changed Files: test/java/net/yacy/document/parser/htmlParserTest.java |
Sun Nov 05 00:52:14 CET 2017 by reger | upd to pdfbox-2.0.8.jar Changed Files: .classpath, build.xml, lib/fontbox-2.0.8.License, lib/fontbox-2.0.8.jar, lib/pdfbox-2.0.8.License, lib/pdfbox-2.0.8.jar, pom.xml |
Sat Nov 04 11:06:05 CET 2017 by luccioman | Renamed Chinese & Greek lng files using ISO639-1 codes. Previously named with their ISO 3166-1 country code : this way, when setting language to "Browser" in ConfigBasic.html, it didn't work properly when browser preferred language was Chinese or Greek as their respective language codes are "zh" and "el" (not "cn" and "gr" which are their country codes) Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java, locales/el.lng, locales/zh.lng, source/net/yacy/data/Translator.java |
Fri Nov 03 10:34:36 CET 2017 by luccioman | Added a help link to ISO 639-1 language codes list ref Changed Files: htroot/index.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Thu Nov 02 08:57:00 CET 2017 by luccioman | Added description of spatial restrictions in search options Changed Files: htroot/index.html, locales/fr.lng, locales/master.lng.xlf |
Tue Oct 31 08:53:17 CET 2017 by luccioman | Customized Threads with generic name for easier monitoring. Changed Files: source/net/yacy/crawler/RecrawlBusyThread.java, source/net/yacy/document/importer/WarcImporter.java, source/net/yacy/search/Switchboard.java |
Tue Oct 31 08:19:04 CET 2017 by luccioman | Added language HTML attribute to the search home page. Changed Files: htroot/index.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/sk.lng, locales/uk.lng |
Tue Oct 31 07:44:37 CET 2017 by luccioman | Updated search page keyboard shortcuts descriptions. Changed Files: htroot/index.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Mon Oct 30 08:07:59 CET 2017 by luccioman | Use accessible labels for search home page radio buttons. Changed Files: htroot/index.html |
Mon Oct 30 07:38:47 CET 2017 by luccioman | Updated a license header typo. Changed Files: source/net/yacy/crawler/CrawlStarterFromScraper.java |
Fri Oct 27 14:00:30 CEST 2017 by Apply55gx | fix typo Changed Files: source/net/yacy/crawler/CrawlStarterFromScraper.java, source/net/yacy/crawler/FileCrawlStarterTask.java |
Tue Oct 24 09:54:54 CEST 2017 by luccioman | Stay authenticated when going to the search start page. Otherwise, when authenticated as admin and navigating from search results or admin pages to the search start page (/index.html), if nothing is done on that page within HTTP Digest Auth timeout (about 2mn), then search is performed without authentication and so without extended search features. Changed Files: htroot/env/templates/simpleSearchHeader.template, htroot/env/templates/simpleheader.template |
Tue Oct 24 09:34:03 CEST 2017 by luccioman | Use the same top nav bar on index.html and search results. Thus eventually including the same optional login link/status in the search start page than in the results page, for the same convenient login without the need to use the Administration section. Changed Files: htroot/index.html, htroot/index.java, htroot/yacysearch.java |
Mon Oct 23 18:28:11 CEST 2017 by luccioman | Fixed loss of index page form values on 'more options' link click. Restores the behavior introduced eleven years ago (see commit 479861a3cf82e3439f7cdcce3865d3de602d53c3) and lost by mistake 3 years ago (see commit 617dd9c97b5db119a4603190ccedaf7d504b728b), when the click handler started referencing a missing HTML id. Changed Files: htroot/index.html |
Thu Oct 19 09:27:52 CEST 2017 by luccioman | Fixed JPEG snapshot resizing when running on OpenJDK. Resizing JPEG snapshot images through /api/snapshot.jpg failed when running on OpenJDK, but rendered successfully with a Oracle JDK. Details in mantis 772 ( http://mantis.tokeek.de/view.php?id=772 ). Removing any alpha component (useless in snapshot images) from the rendered resized image solves the issue. Changed Files: htroot/api/snapshot.java, source/net/yacy/peers/graphics/EncodedImage.java |
Wed Oct 18 14:17:06 CEST 2017 by luccioman | Updated Java version information on Readme Changed Files: README.md |
Wed Oct 18 07:53:07 CEST 2017 by luccioman | Consistently encode snapshot image with format requested on the API. Previously, calling /api/snapshot.png rendered JPEG encoded images. Changed Files: htroot/api/snapshot.java, source/net/yacy/cora/util/Html2Image.java, test/java/net/yacy/cora/util/Html2ImageTest.java |
Tue Oct 17 09:41:58 CEST 2017 by luccioman | Fixed search result Snapshots link. Previously rendered as a broken URL containing the absolute file path of a snapshot on the search server. Now rendered as a valid URL linking to the /api/snapshot API to provide available snapshot content. Snapshot format is selected among the available ones in the following order of preference : JPG/PNG, PDF, and XML. Changed Files: htroot/ConfigSearchPage_p.html, htroot/yacysearchitem.html, htroot/yacysearchitem.java, locales/fr.lng, locales/master.lng.xlf |
Mon Oct 16 19:45:17 CEST 2017 by luccioman | Fixed pdf2image conversion with imagemagick on PDFs having transparency The target image format (jpeg) doesn't support transparency, so the Html2ImageTest produced unusable black images when ran on a linux machine having imagemagick package installed. Changed Files: source/net/yacy/cora/util/Html2Image.java, test/java/net/yacy/cora/util/Html2ImageTest.java |
Mon Oct 16 17:04:22 CEST 2017 by luccioman | Properly close resources (even on error) on OS and ThreadDump classes. Also updated some JavaDoc and main() function usage message on the same ones. Changed Files: source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/util/OS.java |
Mon Oct 16 09:18:12 CEST 2017 by luccioman | Fixed ProfilingGraph calculation integer overflows and added test class. Complementary to fix proposed in PR #128 by @otteresk. Changed Files: htroot/PerformanceGraph.java, source/net/yacy/peers/graphics/ProfilingGraph.java, source/net/yacy/visualization/ChartPlotter.java, test/java/net/yacy/peers/graphics/ProfilingGraphTest.java |
Wed Oct 11 07:13:28 CEST 2017 by luccioman | Addedd missing parameters to yacysearchtrailer call on JS resort mode Changed Files: htroot/js/yacysort.js |
Mon Oct 09 19:08:39 CEST 2017 by luccioman | Adjusted ResponseHeaderTest to succeed on slow or highly loaded CPU Changed Files: test/java/net/yacy/cora/protocol/ResponseHeaderTest.java |
Mon Oct 09 14:25:43 CEST 2017 by luccioman | Added a Travis build status image to Readme Changed Files: README.md |
Sat Oct 07 06:13:22 CEST 2017 by reger | Adjust tags css style in ConfigSearchPage to equal search page Changed Files: htroot/ConfigSearchPage_p.html |
Fri Oct 06 20:32:28 CEST 2017 by reger | Update deprecated SolrInputDocument.addField() with boost value remove unused SchemaConfiguration.getDate (as it is designed to return only past dates which might be unexpected for general configuration schema) Changed Files: source/net/yacy/cora/federate/solr/SchemaConfiguration.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Thu Oct 05 14:42:05 CEST 2017 by luccioman | Updated Debian optional dependencies with the ones used for snapshots Changed Files: debian/control |
Thu Oct 05 14:22:35 CEST 2017 by luccioman | Exclude eventual maven targets from ant dist task. Changed Files: build.xml |
Thu Oct 05 13:09:11 CEST 2017 by luccioman | Updated travis config : install ghostscript, required for Html2Image Changed Files: .travis.yml |
Thu Oct 05 13:09:11 CEST 2017 by luccioman | Updated travis config : install ghostscript, required for Html2Image Changed Files: .travis.yml, source/net/yacy/cora/util/Html2Image.java |
Thu Oct 05 09:25:02 CEST 2017 by luccioman | Updated Travis jdk version to match current requirements (Java 1.8) Changed Files: .travis.yml |
Wed Oct 04 18:33:09 CEST 2017 by luccioman | Added partial bzip2 stream parsing support and bzipParser Junit test Changed Files: source/net/yacy/document/parser/bzipParser.java, test/java/net/yacy/document/parser/bzipParserTest.java, test/parsertest/umlaute_html_utf8.html.bz2, test/parsertest/umlaute_html_xml_txt_gnu.tbz2, test/parsertest/umlaute_linux.txt.bz2 |
Wed Oct 04 08:41:43 CEST 2017 by luccioman | Fixed RWI distance calculation on multi words search queries. Distance was lost when storing/retrieving references to intermediate result container. Now all JUnit tests are again successfully passing! Changed Files: source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, test/java/net/yacy/kelondro/rwi/ReferenceContainerTest.java |
Mon Oct 02 10:05:57 CEST 2017 by luccioman | Added textual hints to language radio buttons labels As an help and accessible alternative to visual styling marking whether a language is available in browser preferred lang mode. Changed Files: htroot/ConfigBasic.html, htroot/ConfigBasic.java |
Mon Oct 02 09:36:13 CEST 2017 by luccioman | Fixed NullPointerException case on 'Browser' lang selection Occurred when English was the only active language, then making the ConfigBasic.html page unusable until manually modifying the locale.language setting. Changed Files: source/net/yacy/data/Translator.java |
Mon Oct 02 02:51:10 CEST 2017 by reger | fix array out of bounds in YJsonResponseWriter and OpensearchResponsWriter on recreation of image url. Set parameter of indexList2protocolList to required number of images (image_stubs) Situation e.g. image_stub(size=15) but images_protocol(size=12) Changed Files: source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java |
Sat Sep 30 11:58:49 CEST 2017 by otter | prevent integer overflow in chartDot for nodes with a big index Changed Files: source/net/yacy/visualization/ChartPlotter.java |
Sat Sep 30 00:48:54 CEST 2017 by otter | prevent integer overflow in chartLine Changed Files: source/net/yacy/visualization/ChartPlotter.java |
Fri Sep 29 00:26:30 CEST 2017 by reger | Adjust filetype: query modifier parameter to lower case to prevent mismatch on user input with mixed case Internally file extension are always compared lowercase. Changed Files: source/net/yacy/search/query/QueryModifier.java |
Thu Sep 28 09:55:23 CEST 2017 by luccioman | Updated master translation file for ConfigSearchPage_p.html Changed Files: locales/de.lng, locales/fr.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng |
Thu Sep 28 00:46:49 CEST 2017 by reger | Add links to the optional keyword tags of search result If swichted on link (click) to the tag adds the keyword to the search query. If a keyword navigator is active the selected keyword adds or replaces a query keyword: modifier (currently replace was choosen as multiple keywords are not fully supported yet) Changed Files: htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Wed Sep 27 17:51:11 CEST 2017 by luccioman | Added French translation for ConfigSearchPage_p.html Changed Files: locales/fr.lng |
Tue Sep 26 14:58:30 CEST 2017 by luccioman | Added missing accessible labels to ConfigSearchPage_p.html Changed Files: htroot/ConfigSearchPage_p.html, htroot/env/templates/submenuDesign.template |
Tue Sep 26 07:59:44 CEST 2017 by luccioman | Fixed ConfigSearchPage_p HTML validation errors. Validated with Nu Html Checker 17.9.0 Changed Files: htroot/ConfigSearchPage_p.html, htroot/env/templates/footer.template |
Mon Sep 25 15:21:17 CEST 2017 by luccioman | Removed unnecessary max counts init on empty search navigators. Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/search.java, source/net/yacy/cora/federate/FederateSearchManager.java |
Mon Sep 25 14:54:35 CEST 2017 by luccioman | Restrict Search Result Layout modification to HTTP POST only. Changed Files: htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java |
Fri Sep 22 11:00:46 CEST 2017 by luccioman | Improved accessibility of histograms widgets. Added keyboard navigation support and missing WAI-ARIA attributes. Tested with NVDA 2017.3 screenreader on recent major browsers. Changed Files: htroot/ConfigSearchPage_p.html, htroot/HostBrowser.html, htroot/js/accessibleHistogram.js, htroot/jslicense.html, htroot/yacysearchtrailer.html |
Wed Sep 20 07:59:20 CEST 2017 by luccioman | Upgraded JavaScript lib raphael.js from 2.1.3 to 2.2.7 Changed Files: htroot/ConfigSearchPage_p.html, htroot/HostBrowser.html, htroot/js/raphael.min.js, htroot/jslicense.html, htroot/yacysearchtrailer.html |
Mon Sep 18 17:36:07 CEST 2017 by luccioman | Refresh paginations buttons instead of fully rendering each time. This prevent the already displayed pagination buttons to be unresponsive when clicking on them while the rendering JS function is running. Changed Files: htroot/js/yacysearch.js, htroot/js/yacysort.js, htroot/yacysearch.html |
Sun Sep 17 00:29:36 CEST 2017 by reger | update classpath for Eclipse project config to Solr 6.6.1 Changed Files: .classpath |
Sun Sep 17 00:27:04 CEST 2017 by reger | update to Solr 6.6.1 (ant build) Changed Files: build.xml |
Sun Sep 17 00:26:18 CEST 2017 by reger | update to Solr 6.6.1 (maven build) Changed Files: pom.xml |
Sat Sep 16 23:58:17 CEST 2017 by reger | update maven source and compiler plugin to latest version Changed Files: pom.xml |
Sat Sep 16 10:13:09 CEST 2017 by luccioman | Handle JS refreshing of belatedly added search navigators Changed Files: htroot/js/yacysort.js |
Sat Sep 16 09:26:08 CEST 2017 by luccioman | Restrict JS results resorting to authenticated users. Until a more efficient DOM refresh model needing less XHR requests per search is implemented. Changed Files: htroot/ConfigPortal_p.html, htroot/yacysearch.java |
Fri Sep 15 14:23:49 CEST 2017 by luccioman | Added HTML ids to search navigators for a more reliable JS refreshing. Changed Files: htroot/js/yacysort.js, htroot/yacysearchtrailer.html |
Fri Sep 15 12:16:24 CEST 2017 by luccioman | Results JS resort : properly handle results with same ranking value. Changed Files: htroot/env/yacysort.css, htroot/js/yacysort.js |
Fri Sep 15 11:12:23 CEST 2017 by luccioman | Added new graphical setting for browser JS/On demand results resorting. Changed Files: htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java |
Fri Sep 15 09:51:34 CEST 2017 by luccioman | Apply JS resort only when currently relevant : p2p text search Changed Files: htroot/yacysearch.java |
Thu Sep 14 09:36:55 CEST 2017 by luccioman | Do not animate unnecessarily when changing page on JS sorted results. Changed Files: htroot/env/yacysort.css, htroot/js/yacysort.js |
Wed Sep 13 19:03:01 CEST 2017 by luccioman | Prevent unnecessary DOM finds in JS resorting functions. Also removed now unused functions earlierPage() and laterPage(). Changed Files: htroot/js/yacysort.js |
Wed Sep 13 09:03:24 CEST 2017 by luccioman | Stop updating results with JS resorting on server feeds termination Changed Files: htroot/js/yacysort.js |
Wed Sep 13 08:35:15 CEST 2017 by luccioman | Updated the JavaScript license information page Changed Files: htroot/jslicense.html |
Wed Sep 13 08:23:19 CEST 2017 by luccioman | Disabled as default verbose browser console logs in yacysort.js Changed Files: htroot/js/yacysort.js |
Wed Sep 13 08:16:29 CEST 2017 by luccioman | Added missing copyright header to the yacysort.js file Changed Files: htroot/js/yacysort.js |
Wed Sep 13 08:06:11 CEST 2017 by luccioman | Moved the JS resort specific styling to the usual YaCy CSS location Changed Files: htroot/env/yacysort.css, htroot/yacysearch.html |
Wed Sep 13 07:58:05 CEST 2017 by luccioman | Disable manual search results resorting when resorting is done with JS Also added a constant for the js resorting setting key. Changed Files: htroot/yacysearch.java, source/net/yacy/search/SwitchboardConstants.java |
Wed Sep 13 07:41:03 CEST 2017 by luccioman | Trigger js resorting animations using only CSS classes. Also added some more descriptive comments. Changed Files: htroot/js/yacysort.js, htroot/yacysort.css |
Mon Sep 11 20:02:19 CEST 2017 by Ryszard Go? | Javascript re-sorting: Remove potentially breaking display property and reset max-height when animation is finished. Changed Files: htroot/yacysort.css |
Sun Sep 10 17:09:35 CEST 2017 by Ryszard Go? | Javascript re-sorting: replace jQuery show() with css animations Changed Files: htroot/js/yacysort.js, htroot/yacysearch.html, htroot/yacysort.css |
Fri Sep 08 11:16:37 CEST 2017 by luccioman | Added Solr filter queries for audio, video and application domains Inspired from the existing one used on image search, and consistent with post filtering on content domain applied in SearchEvent.addNodes(). These filters are quite simplistic but at least audio, video or application search now return results. Previously, when filtering on these content domains, many results pages (and often even the first page) were empty while the total results count suggested that results should be available. This was because filtering on domain was only applied AFTER requesting Solr indexes. Changed Files: source/net/yacy/search/query/QueryGoal.java, source/net/yacy/search/query/QueryParams.java |
Tue Sep 05 00:51:43 CEST 2017 by reger | update master.lng, IndexExport_p.html text Changed Files: locales/master.lng.xlf |
Sun Sep 03 19:34:48 CEST 2017 by JeremyRand | Javascript re-sorting: optimize the jQuery selectors a little bit. Changed Files: htroot/js/yacysort.js |
Sun Sep 03 20:03:48 CEST 2017 by JeremyRand | Fix numbered page navigation from getting corrupted when statistics() runs. Changed Files: htroot/js/yacysearch.js |
Sun Sep 03 20:09:44 CEST 2017 by JeremyRand | Add UI for numbered page navigation when Javascript re-sorting is enabled. Changed Files: htroot/js/yacysearch.js, htroot/js/yacysort.js, htroot/yacysearch.html |
Mon Apr 03 05:33:10 CEST 2017 by JeremyRand | Fix the sidebar item "Wiki Name Space" with Javascript re-sorting. Changed Files: htroot/js/yacysort.js |
Mon Apr 03 05:18:16 CEST 2017 by JeremyRand | (WIP) Add numbered page navigation when Javascript re-sorting is enabled. TODO: Add UI for selecting the number. Changed Files: htroot/js/yacysort.js |
Mon Apr 03 04:32:09 CEST 2017 by JeremyRand | (WIP) Fix the sidebar when Javascript resorting is in use. TODO: Add some markup so that DOM traversal in the animations is less painful. Changed Files: htroot/js/yacysort.js, htroot/yacysearch.html, htroot/yacysearchtrailer.html |
Sun Sep 03 19:50:08 CEST 2017 by JeremyRand | (WIP) Optionally sort HTML search items via Javascript. TODO: Expose a GUI setting for this. Changed Files: defaults/yacy.init, htroot/js/yacysort.js, htroot/yacysearch.html, htroot/yacysearch.java |
Mon Aug 28 16:33:53 CEST 2017 by JeremyRand | Add data-ranking attribute to each HTML search item. Changed Files: htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Sat Sep 02 09:53:38 CEST 2017 by luccioman | Updated internal ISO 639-1 language codes with latest standards. Includes 54 language code additions, some name modifications, and marking a few deprecated. Changed Files: source/net/yacy/kelondro/util/ISO639.java |
Thu Aug 31 11:24:59 CEST 2017 by luccioman | Fixed count of filtered results from local solr. Was inadequately modified in my previous related commits (making next pages buttons unavailable in Search portal mode), as SearchEvent.local_solr_available did not count the total filtered results but only the ones within the currently fetched result page(s). Changed Files: htroot/yacysearch.java, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, source/net/yacy/search/query/SearchEvent.java |
Wed Aug 30 23:50:14 CEST 2017 by Michael Peter Christen | try to fix problem with error description http://forum.yacy-websuche.de/viewtopic.php?f=5&t=6023&p=33889&sid=37bc7aa029422be571b9266cdef43c52#p33889 Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Wed Aug 30 12:23:45 CEST 2017 by luccioman | Use local solr filtered results in total search results count. This modification has indeed low incidence as eventual query modifiers are already applied when requesting the local solr index. It mainly impact doublons detected with results from remote peers. Also updated javadocs for clarification. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Aug 29 08:16:12 CEST 2017 by luccioman | Make result action links visible when focusing them with keyboard. Changed Files: htroot/env/base.css |
Tue Aug 29 07:39:12 CEST 2017 by luccioman | Removed duplicate HTML class attribute. Changed Files: htroot/yacysearch.html |
Mon Aug 28 19:03:51 CEST 2017 by luccioman | Added a button to manually refresh sorting of p2p search results. As a server-side oriented alternative to the JavaScript realtime resorting feature proposed in PR #104. The goal is the same as in this PR : having the possibility compensate the network latency of various peers results fetching and obtain once possible a consistently ranked result set. Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html, htroot/yacysearch.java, source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java, source/net/yacy/search/query/SearchEvent.java |
Sun Aug 27 04:22:39 CEST 2017 by reger | update master.lng, RankingSolr_p.html text Changed Files: locales/master.lng.xlf |
Wed Aug 23 08:20:37 CEST 2017 by luccioman | Use Javadoc style comments on SearchEvent properties. For better code readability and understanding. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Aug 22 14:13:00 CEST 2017 by luccioman | Added unit tests on the gzip parser. Changed Files: source/net/yacy/document/parser/gzipParser.java, test/java/net/yacy/document/parser/gzipParserTest.java, test/parsertest/umlaute_html_utf8.html.gz, test/parsertest/umlaute_html_xml_txt_gnu.tgz, test/parsertest/umlaute_linux.txt.gz |
Tue Aug 22 14:11:35 CEST 2017 by luccioman | Finer control on max links to parse in the html parser. Changed Files: source/net/yacy/cora/storage/SizeLimitedMap.java, source/net/yacy/cora/storage/SizeLimitedSet.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/htmlParser.java, test/java/net/yacy/document/parser/htmlParserTest.java, test/parsertest/umlaute_html_namedentities.html |
Tue Aug 22 14:06:09 CEST 2017 by luccioman | Added some unit tests on FileUtils. Changed Files: test/java/net/yacy/kelondro/util/FileUtilsTest.java |
Sun Aug 20 22:17:27 CEST 2017 by reger | Allow to stop currently running warc import (stop button) Changed Files: htroot/IndexImportWarc_p.html, htroot/IndexImportWarc_p.java, source/net/yacy/document/importer/WarcImporter.java |
Wed Aug 16 14:21:07 CEST 2017 by luccioman | Use unredirected robots.txt URL when adding an entry to the table. Changed Files: source/net/yacy/crawler/robots/RobotsTxt.java |
Wed Aug 16 09:30:33 CEST 2017 by luccioman | Ensure proper synchronous robots entry retrieval on first check. Previously, when checking for the first time the robots.txt policy on a unknown host (not cached in the robots table), result was always empty in the /getpageinfo_p.xml api and in the /CrawlCheck_p.html page. Next calls returned however the correct information. Changed Files: htroot/api/getpageinfo_p.java, source/net/yacy/crawler/robots/RobotsTxt.java |
Tue Aug 15 21:04:36 CEST 2017 by luccioman | Upgraded Docker base image from deprecated java to openjdk. Changed Files: docker/Dockerfile, docker/Dockerfile.alpine |
Tue Aug 15 10:11:05 CEST 2017 by luccioman | Prevent search result failure on incomplete images information. Complements the recent modification related to images in commit 7f395ef. Unfortunately many documents metadata fetched from the freeworld p2p network have only partial information about embedded images. Without proper error handling, this made many searches in p2p mode to fail completely. Changed Files: htroot/yacysearchitem.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Tue Aug 15 07:16:01 CEST 2017 by Michael Peter Christen | added usage of X-Real-IP http header to identify request IPs which came through NGINX reverse proxy configurations Changed Files: source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/http/servlets/SolrSelectServlet.java |
Mon Aug 14 20:12:09 CEST 2017 by Michael Peter Christen | added image link in search results This should be a help to make a preview of search results. The image is computed from the list of embedded images, it is always the first image in that list. In rss-type results the image is presented like <media:content medium="image" url="https://abc.xyz/logo.png"/> as defined in http://www.rssboard.org/media-rss#media-content Changed Files: htroot/yacysearchitem.java, htroot/yacysearchitem.json, htroot/yacysearchitem.xml, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/YJsonResponseWriter.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Mon Aug 14 14:47:01 CEST 2017 by luccioman | Also handle text content when parsing XML within limits. Changed Files: source/net/yacy/document/parser/GenericXMLParser.java, test/java/net/yacy/document/parser/GenericXMLParserTest.java |
Mon Aug 14 02:16:43 CEST 2017 by reger | Add junit test for AbstractOperations.addOperand() Changed Files: test/java/net/yacy/cora/federate/solr/logic/AbstractOperationsTest.java |
Mon Aug 14 01:03:15 CEST 2017 by reger | Correction of https://github.com/yacy/yacy_search_server/commit/d03e2c98ea6bd5701c8e8257174c439b9c006afb Fix Conjunction.addOperator to do nothing if term is empty prevent to result in query string with repeated logical operator like "field:term AND AND field:term" possibliy causing out of mem in postprocessing_doublecontent Changed Files: source/net/yacy/cora/federate/solr/logic/AbstractOperations.java |
Mon Aug 14 00:52:03 CEST 2017 by reger | Fix Conjunction.addOperator to do nothing if term is empty prevent to result in query string with repeated logical operator like "field:term AND AND field:term" possibliy causing out of mem in postprocessing_doublecontent Changed Files: source/net/yacy/cora/federate/solr/logic/AbstractOperations.java |
Sat Aug 12 21:53:04 CEST 2017 by reger | Remove deprecated YaCyProxyServlet was replaced by UrlProxyServlet Changed Files: defaults/web.xml |
Sat Aug 12 09:43:49 CEST 2017 by luccioman | Prevent unwanted cached bytes duplication on stream parsing. Changed Files: source/net/yacy/document/TextParser.java |
Sat Aug 12 09:42:06 CEST 2017 by luccioman | Updated xml parser limited parsing test for use latest jdk. Changed Files: test/java/net/yacy/document/parser/GenericXMLParserTest.java |
Fri Aug 11 20:34:59 CEST 2017 by luccioman | Updated debian package configuration to match new Java 1.8 target Following migration from Java 1.7 to Java 1.8 in commit 6fe735945da97abcbb91ac545fb11cff9d48effc Changed Files: debian/control |
Thu Aug 10 23:57:37 CEST 2017 by reger | upde to icu4j-59_1.jar Changed Files: .classpath, build.xml, lib/icu4j-59_1.jar, pom.xml |
Sun Aug 06 23:41:53 CEST 2017 by reger | Skip public post of jre version. Added to determine switch to java8 https://github.com/yacy/yacy_search_server/commit/596b5dfa5936b25b605c42807730c29a1d08cd15 Changed Files: htroot/Network.html, htroot/Network.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Sun Aug 06 23:26:27 CEST 2017 by reger | Replace deprecated ConcurrentHashSet with recommended Java8 ConcurrentHashMap.newKeySet() in postprocessDocuments() Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Sat Aug 05 23:47:27 CEST 2017 by reger | Harmonizing use of xml reader / sax parser in XMLBlacklistImporter eliminating the need for lib/xercesImpl.jar Changed Files: .classpath, build.xml, pom.xml, source/net/yacy/data/list/XMLBlacklistImporter.java |
Sat Aug 05 22:30:06 CEST 2017 by reger | Patch last_modified date with internal FirstSeenTime() if no date provided to make sure updated documents are indexed with their last-modified date as provided in current crawl. (to patch moddate always with firstseen might bear the risk of miss actual updates). Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Tue Aug 01 00:59:53 CEST 2017 by reger | Remove obsolete Protocol parameter ttl (time to live) not interpreted in target yacy/query.html also Protocol.querySeed() not used and parameter not interpreted in target servlet yacy/query.html Changed Files: source/net/yacy/peers/Protocol.java |
Mon Jul 31 23:38:10 CEST 2017 by reger | upd to poi-3.16.jar Changed Files: .classpath, build.xml, lib/poi-3.16.License, lib/poi-3.16.jar, lib/poi-scratchpad-3.16.jar, pom.xml |
Mon Jul 31 01:55:01 CEST 2017 by reger | Replace deprecated getIP with getIPs in Protocol transferURL() and getProfile(). Remember used ip for error handling and departInterface Changed Files: source/net/yacy/peers/Protocol.java |
Sun Jul 30 23:02:15 CEST 2017 by reger | Replace one more deprecated peerDeparture in Protocol.transferIndex() by moving/using interfaceDeparture() in transferRWI() Changed Files: source/net/yacy/peers/Protocol.java |
Sun Jul 30 20:09:06 CEST 2017 by reger | upd to pdfbox-2.0.7.jar Changed Files: .classpath, build.xml, lib/fontbox-2.0.7.License, lib/fontbox-2.0.7.jar, lib/pdfbox-2.0.7.License, lib/pdfbox-2.0.7.jar, pom.xml |
Sun Jul 23 03:55:56 CEST 2017 by reger | Add SolrConfig ClassicIndexSchemaFactory to prevent Solr startup warning. This overrides Solr default to use managed schema. As we don't use programatic schema changes this directs Solr to use schema.xml, eliminating the warning. Changed Files: defaults/solr/solrconfig.xml |
Mon Jul 17 15:35:10 CEST 2017 by luccioman | Log an error when Solr folder migration fails for some reason. Changed Files: source/net/yacy/search/index/Fulltext.java |
Sun Jul 16 23:37:28 CEST 2017 by reger | upd to jwat-warc-1.1.0.jar Changed Files: .classpath, build.xml, lib/jwat-archive-common-1.1.0.jar, lib/jwat-common-1.1.0.jar, lib/jwat-gzip-1.1.0.jar, lib/jwat-warc-1.1.0.jar, pom.xml |
Sun Jul 16 23:35:56 CEST 2017 by reger | upd version for typeahead.jquery.js in jslicense.html Changed Files: htroot/jslicense.html |
Sun Jul 16 14:46:46 CEST 2017 by luccioman | Support parsing gzip files from servers with redundant headers. Some web servers provide both 'Content-Encoding : "gzip"' and 'Content-Type : "application/x-gzip"' HTTP headers on their ".gz" files. This was annoying to fail on such resources which are not so uncommon, while non conforming (see RFC 7231 section 3.1.2.2 for "Content-Encoding" header specification https://tools.ietf.org/html/rfc7231#section-3.1.2.2) Changed Files: source/net/yacy/crawler/retrieval/StreamResponse.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/gzipParser.java |
Sun Jul 16 14:37:06 CEST 2017 by luccioman | URL Viewer : apply crawler size limits when adding to local index. This allow large files parsing and preview, while preventing unwanted OutOfMemory errors which are likely to occur when adding to the Solr Index resources larger than configured crawler limits. Changed Files: htroot/ViewFile.java |
Sat Jul 15 00:19:23 CEST 2017 by reger | Clean up unmaintained and unused AugmentParser trail. Changed Files: |
Fri Jul 14 23:41:39 CEST 2017 by reger | Clean up redundant but obsolete jquery.rdfquery-core-1.0.js script lib Changed Files: htroot/jslicense.html |
Thu Jul 13 08:18:40 CEST 2017 by luccioman | Added gzip parser support for max content bytes limit Changed Files: source/net/yacy/document/parser/gzipParser.java |
Thu Jul 13 08:12:10 CEST 2017 by luccioman | Added HTML parser support for maximum content bytes parsing limit Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/htmlParser.java |
Wed Jul 12 16:03:23 CEST 2017 by luccioman | Merge pull request #122 from Scarfmonster/patch-1 I also reproduced the issue, and the fix is working fine. Thanks @Scarfmonster Changed Files: source/net/yacy/http/Jetty9HttpServerImpl.java |
Wed Jul 12 00:18:12 CEST 2017 by luccioman | Added RSS parser support for maximum content bytes parsing limit Changed Files: source/net/yacy/cora/document/feed/RSSFeed.java, source/net/yacy/cora/document/feed/RSSReader.java, source/net/yacy/document/Document.java, source/net/yacy/document/parser/rssParser.java |
Wed Jul 12 00:13:24 CEST 2017 by luccioman | Finer control on bounded input streams with custom stream implementation Changed Files: source/net/yacy/cora/util/StreamLimitException.java, source/net/yacy/cora/util/StrictLimitInputStream.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/GenericXMLParser.java |
Tue Jul 11 09:07:48 CEST 2017 by luccioman | Added parsing within bounds implementation to the generic parser. Changed Files: source/net/yacy/document/parser/genericParser.java |
Tue Jul 11 09:06:37 CEST 2017 by luccioman | Support trying multiple parsers even when streaming on large resources. Changed Files: source/net/yacy/document/TextParser.java |
Tue Jul 11 09:04:23 CEST 2017 by luccioman | Support loading local files with a per request specified maximum size. Consistently with the HTTP loader implementation. Changed Files: source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/repository/LoaderDispatcher.java |
Sun Jul 09 23:08:54 CEST 2017 by reger | Fix css conflict of YMarks.html to make it viewable. yacy-ymarks.css sidebar conflicts with bootstraps sidebar (different overlay settings). Simply renamed it to ymark-sidebar. Changed Files: htroot/YMarks.html, htroot/env/yacy-ymarks.css |
Sat Jul 08 23:46:10 CEST 2017 by reger | upd to commons-fileupload-1.3.3.jar Changed Files: .classpath, build.xml, lib/commons-fileupload-1.3.3.License, lib/commons-fileupload-1.3.3.jar, pom.xml |
Mon Jul 03 14:53:36 CEST 2017 by luccioman | Removed temporary html parser test code Changed Files: test/java/net/yacy/document/parser/htmlParserTest.java |
Mon Jul 03 13:51:14 CEST 2017 by luccioman | URL Viewer : decode raw text using the eventual response charset. When provided, or decode as UTF-8 as previously done. Changed Files: htroot/ViewFile.java |
Mon Jul 03 10:00:53 CEST 2017 by luccioman | HTML parser : removed unnecessary remaining recursive processing Recursive processing was removed in commit 67beef657f82e92f48dd8425073ad81896a2ff4b, but one remained for anchors content(likely omitted from refactoring). It is no more necessary : other links such as images embedded in anchors are currently correctly detected by the parser. More annoying : that remaining recursive processing could lead to almost endless processing when encountering some (invalid) HTML structures involving nested anchors, as detected and reported by lucipher on YaCy forum ( http://forum.yacy-websuche.de/viewtopic.php?f=23&t=6005 ). Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Fri Jun 30 11:41:48 CEST 2017 by luccioman | Updated PerformanceQueues_p.xml API with last related servlet changes Changed Files: htroot/PerformanceQueues_p.xml |
Fri Jun 30 11:30:54 CEST 2017 by luccioman | Made remote search max system load limits configurable from UI. As reported by davide on YaCy forums ( http://forum.yacy-websuche.de/viewtopic.php?f=23&t=6004 ) when the system is on high load, unless reading carefully YaCy configuration file, it could be difficult to understand why remote search results are not fetched. Changed Files: htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/SwitchboardConstants.java |
Fri Jun 30 02:11:18 CEST 2017 by reger | Add keyword constraint to rwi query result filter To discard rwi results not matching query keyword: parameter Changed Files: source/net/yacy/search/query/SearchEvent.java |
Fri Jun 30 01:13:47 CEST 2017 by luccioman | Apply consistent behavior on HTTP resource size exceeding limit. On content size known from HTTP headers, terminates connection faster and improves error reports quality by reporting relevant message "Content to download exceed maximum value..." rather than previously "no response (NULL) for url...". Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Fri Jun 30 00:30:54 CEST 2017 by luccioman | Respect maxFileSize limit also when streaming HTTP and when relevant. Constraint applied consistently with HTTP content full load in byte array. Changed Files: source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/visualization/ImageViewer.java |
Thu Jun 29 11:36:47 CEST 2017 by luccioman | Added an informative title on the crawl start robots.txt status icon Changed Files: htroot/js/IndexCreate.js |
Thu Jun 29 11:25:27 CEST 2017 by luccioman | Crawl start Ajax request : properly handle eventual XML parsing errors Otherwise on a malformed getpageinfo_p XML response (from the browser point of view), JavaScript errors where thrown and the ajax status steering wheel remained displayed indefinitely. Changed Files: htroot/js/IndexCreate.js |
Tue Jun 27 19:30:40 CEST 2017 by luccioman | Refactored plain-text URLs detection implementation. For faster processing (measured about 2 times faster on many real-world examples) and more advanced detection (previous algorithm detected only URLs separated from the rest of the text by a space character). Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, test/java/net/yacy/document/parser/html/ContentScraperTest.java |
Mon Jun 26 17:33:56 CEST 2017 by luccioman | Made mime type and extension normalization locale independent. Previously, upper cased mime type was incorrectly normalized when the default locale is Turkish. Changed Files: source/net/yacy/document/TextParser.java, test/java/net/yacy/document/TextParserTest.java |
Sun Jun 25 20:05:37 CEST 2017 by reger | upd to jwat-warc-1.0.6.jar Changed Files: .classpath, build.xml, lib/jwat-archive-common-1.0.6.jar, lib/jwat-common-1.0.6.jar, lib/jwat-gzip-1.0.6.jar, lib/jwat-warc-1.0.6.jar, pom.xml |
Sat Jun 24 23:15:25 CEST 2017 by reger | remove unused Solr optional extra handler lib solr-dataimporthandler-6.6.0.jar Changed Files: .classpath, build.xml |
Sat Jun 24 22:54:43 CEST 2017 by reger | upd to jsoup-1.10.3.jar Changed Files: .classpath, build.xml, lib/jsoup-1.10.3.jar, pom.xml |
Fri Jun 23 02:23:49 CEST 2017 by Ryszard Go? | Wrong password was removed after the SSL certificate import Removing the keystore password will prevent ssl from working after the next restart. The certificate password should be removed instead. Fixes http://mantis.tokeek.de/view.php?id=687 Changed Files: source/net/yacy/http/Jetty9HttpServerImpl.java |
Thu Jun 22 10:50:34 CEST 2017 by luccioman | Improved character encoding detection from Content-Type header Also updated some related JavaDocs Changed Files: source/net/yacy/cora/protocol/HeaderFramework.java, test/java/net/yacy/cora/protocol/HeaderFrameworkTest.java |
Wed Jun 21 09:14:50 CEST 2017 by luccioman | Added a basic JUnit test with test gz files for the gzip parser Changed Files: test/java/net/yacy/document/parser/gzipParserTest.java, test/parsertest/umlaute_html_utf8.html.gz, test/parsertest/umlaute_linux.txt.gz |
Wed Jun 21 09:11:17 CEST 2017 by luccioman | Properly close test files in htmlParser unit test Changed Files: test/java/net/yacy/document/parser/htmlParserTest.java |
Mon Jun 19 17:02:11 CEST 2017 by luccioman | Prevent integer overflow in table statistics and use strong typing Changed Files: htroot/PerformanceMemory_p.java, source/net/yacy/kelondro/table/Table.java |
Sat Jun 17 09:33:14 CEST 2017 by luccioman | Limit the number of initially previewed links in crawl start pages. This prevent rendering a big and inconvenient scrollbar on resources containing many links. If really needed, preview of all links is still available with a "Show all links" button. Doesn't affect the number of links used once the crawl is effectively started, as the list is then loaded again server-side. Changed Files: htroot/CrawlStartExpert.html, htroot/CrawlStartSite.html, htroot/api/getpageinfo_p.java, htroot/api/getpageinfo_p.xml, htroot/js/IndexCreate.js |
Sat Jun 17 09:26:37 CEST 2017 by luccioman | Improved stream-oriented parsing entering conditions. Changed Files: source/net/yacy/document/TextParser.java |
Fri Jun 16 08:50:57 CEST 2017 by luccioman | Limit scope of some local JavaScript variables. Changed Files: htroot/js/IndexCreate.js |
Fri Jun 16 08:44:40 CEST 2017 by Michael Peter Christen | added json(p) endpoint for crawl start Changed Files: htroot/Crawler_p.java, htroot/Crawler_p.json |
Fri Jun 16 06:31:45 CEST 2017 by reger | make nsis build script require java 8 Changed Files: build.nsi |
Fri Jun 16 02:17:49 CEST 2017 by reger | update nsi installer java autodl bundleid to use jre-8u131 Changed Files: build.nsi |
Fri Jun 16 00:12:09 CEST 2017 by reger | remove reference to velocityresponsewriter in solrconfig.xml it is not longer part of solr-core api http://lucene.apache.org/solr/6_6_0/index.html Changed Files: defaults/solr/solrconfig.xml |
Thu Jun 15 21:02:18 CEST 2017 by reger | remove sample path setting in solrconfig.xml not valid in Yacy resulting in startup stop exception after fresh swithch to 1.921 Changed Files: defaults/solr/solrconfig.xml |
Thu Jun 15 20:24:53 CEST 2017 by reger | update maven pom setting to YaCy version 1.921 java 1.8 and solr 6.6 Changed Files: pom.xml |
Thu Jun 15 14:13:46 CEST 2017 by luccioman | Prevent high CPU load at startup, caused by the Solr suggester build. Reported by Collision on mantis 758 ( http://mantis.tokeek.de/view.php?id=758 ). Introduced by the new YaCy Solr configuration for Solr 6.6.0 (see commit 6fe735945da97abcbb91ac545fb11cff9d48effc), including now Suggester configuration. Changed Files: defaults/solr/solrconfig.xml |
Thu Jun 15 09:50:02 CEST 2017 by luccioman | Added HT Cache basic statistics (hit rate) Changed Files: htroot/ConfigHTCache_p.html, htroot/ConfigHTCache_p.java, source/net/yacy/crawler/data/Cache.java, test/java/net/yacy/crawler/data/CacheTest.java |
Thu Jun 15 09:48:22 CEST 2017 by luccioman | Use volatile to ensure concurrent threads use up to date property value Changed Files: source/net/yacy/kelondro/blob/Compressor.java |
Wed Jun 14 19:02:08 CEST 2017 by luccioman | Made Cache compression level and lock timeout user configurable Changed Files: defaults/yacy.init, htroot/ConfigHTCache_p.html, htroot/ConfigHTCache_p.java, source/net/yacy/crawler/data/Cache.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, test/java/net/yacy/crawler/data/CacheTest.java |
Wed Jun 14 08:56:11 CEST 2017 by luccioman | Prevent log pollution from unwanted Solr warnings. Many non-blocking "java.nio.file.NoSuchFileException" traces with warning log level can be logged by Solr, especially when heavily crawling. This is issue is known from Solr 5.x but still unresolved with Solr 6.x ( https://issues.apache.org/jira/browse/SOLR-9120 ) Consequently upgraded to "SEVERE" the default log level of the related internal Solr class. See also mantis 727 ( http://mantis.tokeek.de/view.php?id=727 ) Changed Files: defaults/yacy.logging |
Fri Jun 09 12:50:36 CEST 2017 by Michael Peter Christen | re-added solr synchronization hack Changed Files: source/net/yacy/cora/federate/solr/connector/SolrServerConnector.java |
Thu Jun 08 07:36:11 CEST 2017 by luccioman | Ensure system resource release by closing document stream. Changed Files: source/net/yacy/document/TextParser.java |
Tue Jun 06 10:30:02 CEST 2017 by luccioman | Removed unnecessary finalize implementation. On such private classes with limited scope but with frequent instance creations and removals within the application lifecycle, implementing the finalize method is particularly unwanted as it decreases the garbage collector performance. What's more the Object.finalize() method is now deprecated in the JDK 9 and will eventually disappear from future releases (see https://bugs.openjdk.java.net/browse/JDK-8177970) Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java |
Sun Jun 04 01:50:40 CEST 2017 by reger | Tokenize result entry keywords and add some styling for display Changed Files: htroot/env/base.css, htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Sat Jun 03 21:58:04 CEST 2017 by reger | upd to commons-compress-1.14.jar Changed Files: .classpath, build.xml, lib/commons-compress-1.14.License, lib/commons-compress-1.14.jar, pom.xml |
Fri Jun 02 09:47:45 CEST 2017 by luccioman | Unsure closing ChunkIterator stream in every possible use case. Also trace in logs the eventual close failures instead of failing silently. This should help prevent holding too many unreleased system file handlers, as in the case reported by eros on YaCy forum (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5988&sid=b00e7486c1bf7e48a0d63eb328ccca02 ) Changed Files: source/net/yacy/kelondro/table/ChunkIterator.java, source/net/yacy/kelondro/table/Table.java |
Fri Jun 02 01:46:06 CEST 2017 by luccioman | Improved consistency between loader openInputStream and load functions Changed Files: source/net/yacy/crawler/retrieval/FTPLoader.java, source/net/yacy/crawler/retrieval/FileLoader.java, source/net/yacy/crawler/retrieval/HTTPLoader.java, source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/crawler/retrieval/SMBLoader.java, source/net/yacy/crawler/retrieval/StreamResponse.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/visualization/ImageViewer.java |
Tue May 30 17:38:16 CEST 2017 by luccioman | Added JavaDoc to the getpageinfo_p API servlet. Changed Files: htroot/api/getpageinfo_p.java |
Tue May 30 09:29:28 CEST 2017 by luccioman | Deprecated duplicated and internally unused getpageinfo servlet. Redirections set for the transition of any eventual external uses: - /api/getpageinfo.xml to /api/getpageinfo_p.xml - /api/getpageinfo.json to /api/getpageinfo_p.json Changed Files: htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.json |
Mon May 29 19:16:09 CEST 2017 by luccioman | Fixed a NullPointerException case on Digest authentication. Could occur when upgrading from a Debian package configured with Basic authentication (as in release 1.92.9000) to a more recent one with Digest authentication, without having re-encoded the admin password (for example with dpkg-reconfigure). As reported by eros on YaCy forum (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5988#p33686). Changed Files: source/net/yacy/http/YaCyLegacyCredential.java |
Wed May 24 22:13:42 CEST 2017 by reger | upd to pdfbox-2.0.6.jar Changed Files: .classpath, build.xml, lib/fontbox-2.0.6.License, lib/fontbox-2.0.6.jar, lib/pdfbox-2.0.6.License, lib/pdfbox-2.0.6.jar, pom.xml |
Wed May 24 08:43:03 CEST 2017 by luccioman | Quoted param value in Solr query to avoid unwanted traces in logs When Webgraph Solr core is enabled, crawling and removing from index an URL whose hash starts with the '-' character (example URL : https://cs.wikipedia.org/ whose hash is "-2-HuTEndn4x") produced a full ParseException stack trace in YaCy logs. This was not blocking because the Solr query parser is able to escape itself the query and run it successfully, but filled uselessly YaCy logs. Changed Files: source/net/yacy/search/index/Fulltext.java |
Tue May 23 07:25:40 CEST 2017 by luccioman | Restored search page default behavior for Tab, Page Up and Down keys Replaced by shortcuts defined by the HTML "accesskey" attribute which has the advantage to be advertised by screen readers when focusing the corresponding buttons, contrary to custom JavasScript key handlers. Now With Firefox : - "Alt + Shift + n" for next page - "Alt + Shift + p" for previous page Following ARIA recommendation : "keyboard shortcuts enhance, not replace, standard keyboard access." ( see https://www.w3.org/TR/wai-aria-practices/#kbd_shortcuts_behavior_design) Fix for mantis 711 (http://mantis.tokeek.de/view.php?id=711) Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html |
Mon May 22 01:56:11 CEST 2017 by reger | Set request originator to own peer in warc importer in addition to change in https://github.com/yacy/yacy_search_server/commit/039162fbf0eca808afd350d360c3bcfe62dc4195 Changed Files: source/net/yacy/document/importer/WarcImporter.java |
Mon May 22 01:34:08 CEST 2017 by reger | Change warc importer to use defaultsurrogate-crawl profile, as reported by LA_FORGE http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5990 and analysed by @luccioman (see comment https://github.com/yacy/yacy_search_server/commit/510f11d3745e14841420781376b733fd248d51f3) it creates conflict using a other crawlprofile without setting originator. Changed Files: source/net/yacy/document/importer/WarcImporter.java |
Thu May 18 00:28:00 CEST 2017 by Michael Peter Christen | added a cache to prevent too many seed enumerations Changed Files: source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java |
Wed May 17 09:00:29 CEST 2017 by luccioman | Enable p2p and cluster communication when "Protection of all pages" on As reported by paul89 on YaCy forum (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5958 ), when setting the "Protection of all pages" to "On" in the "ConfigAccounts_p.html" page, the peer became completely unreachable by others, which is not the purpose of this feature. But the restriction still makes sense as a security enforcement and is maintained in private "Robinson mode" where by the way any peer-to-peer or cluster communication would be rejected. Changed Files: source/net/yacy/http/Jetty9YaCySecurityHandler.java |
Tue May 16 09:44:13 CEST 2017 by luccioman | Added missing accessibility attributes on search results progress bar. Changed Files: htroot/js/yacysearch.js, htroot/yacysearch.html |
Mon May 15 13:31:24 CEST 2017 by luccioman | Annotated search result information separators for screen readers. Changed Files: htroot/ConfigSearchPage_p.html, htroot/yacysearchitem.html |
Sat May 13 20:38:25 CEST 2017 by sgaebel | added closing of lst-Tag in solr-Export Changed Files: source/net/yacy/search/index/Fulltext.java |
Thu May 11 08:33:19 CEST 2017 by luccioman | Added some JavaDoc Changed Files: source/net/yacy/peers/RemoteSearch.java |
Tue May 09 22:52:54 CEST 2017 by reger | Adjust mergeDocuments to keep youngest last-modified date of document collection Changed Files: source/net/yacy/document/Document.java, test/java/net/yacy/document/DocumentTest.java |
Tue May 09 18:32:47 CEST 2017 by luccioman | Fixed StringIndexOutOfBoundsException case. Revealed by commit c77e43a : the exception was then thrown when indexing pages containing mailto: scheme URL links with the Solr Webgraph core enabled. Fixed the error case and restored filtering on mailto links in Document.resortLinks() as these URLs still should not appear in Document.hyperlinks. Changed Files: source/net/yacy/document/Document.java, source/net/yacy/search/schema/WebgraphConfiguration.java |
Tue May 09 12:20:41 CEST 2017 by luccioman | Updated Debian package post install script admin password encoding. To fit the now default HTTP authentication method set to Digest in commit f7fce1b. Also fixed unauthenticated access from localhost setting when first installing the Debian package and letting the prompted password field empty. Changed Files: debian/postinst |
Thu May 04 16:36:45 CEST 2017 by luccioman | Improved new blacklist entries URL scheme detection. Changed Files: source/net/yacy/repository/BlacklistHelper.java, test/java/net/yacy/repository/BlacklistHelperTest.java |
Thu May 04 11:21:27 CEST 2017 by luccioman | Updated putHTML() JavaDoc Changed Files: source/net/yacy/server/serverObjects.java |
Thu May 04 11:19:59 CEST 2017 by luccioman | Handle '?' and '+' chars as valid wild cards when adding to blacklist. An entry such as "domain.com/[a-z]+" is a valid regular expression and do not need additional ".*.*/.*" wildcards. Changed Files: source/net/yacy/repository/BlacklistHelper.java |
Thu May 04 11:12:58 CEST 2017 by luccioman | Fixed blacklist Regex containing '+' characters rendering. As reported on YaCy forum by shni (http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5970) when a blacklist entry contained both '?' and '+' characters, the '+' chars were wrongly decoded and rendered as spaces. Changed Files: htroot/Blacklist_p.java |
Wed May 03 18:53:01 CEST 2017 by luccioman | Added MediaWiki dump import scheduling feature. Checking the last modified date by default to prevent unnecessary long running operations. Changed Files: htroot/IndexImportMediawiki_p.html, htroot/IndexImportMediawiki_p.java, source/net/yacy/data/WorkTables.java |
Tue May 02 09:38:45 CEST 2017 by luccioman | Improved MediaWiki dump import monitoring. When import thread is terminated : - now stop refreshing and stay on the monitoring page to give user a feedback after a long running import - added link to the next monitoring step : results from surrogates reader - added link to new import On the new import page, added a link on the eventual last import report. Changed Files: htroot/IndexImportMediawiki_p.html, htroot/IndexImportMediawiki_p.java |
Tue May 02 09:33:11 CEST 2017 by luccioman | Added some JavaDoc Changed Files: source/net/yacy/document/importer/Importer.java |
Tue May 02 09:32:04 CEST 2017 by luccioman | Fixed regression introduced by commit 9ad4d16 On MediaWiki dump imports, the SurrogateReader was trying to unread too many bytes, then failing with the following exception : "java.io.IOException: Push back buffer is full". Changed Files: source/net/yacy/document/content/SurrogateReader.java |
Mon May 01 11:38:02 CEST 2017 by Michael Peter Christen | added patch to rewrite altered yacy grid schema into yacy schema This generates the stub and protocol parts of an url for inboundlinks, outboundlinks and images Changed Files: source/net/yacy/search/Switchboard.java |
Sun Apr 30 23:53:52 CEST 2017 by reger | Add a responsHeader to the solr index export with a format identifier and export parameter (in accordance with response xml format) for easier format detection on import. Changed Files: source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, source/net/yacy/search/index/Fulltext.java |
Fri Apr 28 11:39:51 CEST 2017 by luccioman | Fixed Index Export feature for compatibility with old indexed documents. This is a fix for mantis 682 (http://mantis.tokeek.de/view.php?id=682) and issue #116 Changed Files: source/net/yacy/search/index/Fulltext.java |
Fri Apr 28 11:36:48 CEST 2017 by luccioman | Added some JavaDoc Changed Files: source/net/yacy/cora/federate/solr/SchemaDeclaration.java |
Thu Apr 27 18:24:54 CEST 2017 by luccioman | Crawl results page : apply table lines number limit. Take into account the already existing default limit value (especially useful after a long crawl or surrogates import), or a custom one from parameter "count". Added a "Show all" link for convenience. Changed Files: htroot/CrawlResults.html, htroot/CrawlResults.java |
Thu Apr 27 09:50:04 CEST 2017 by luccioman | Extended WikiCode template inclusion syntax support. Wiki templates are not rendered but syntax support is improved, which greatly enhance snippets rendering on search results coming from a MediaWiki dump import. Tested on various dumps from Wikimedia at https://dumps.wikimedia.org/backup-index.html See also Wikipedia transclusion documentation at https://en.wikipedia.org/wiki/Wikipedia:Transclusion Changed Files: source/net/yacy/data/wiki/WikiCode.java, test/java/net/yacy/data/wiki/WikiCodeTest.java |
Tue Apr 25 08:44:02 CEST 2017 by Michael Peter Christen | added yacy grid flatjson surrogate parser Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/search/schema/CollectionSchema.java |
Mon Apr 24 18:24:26 CEST 2017 by luccioman | Fixed surrogates import monitoring page (/CrawlResults.html?process=7) This page was always empty, as described in mantis 740 (http://mantis.tokeek.de/view.php?id=740) Changed Files: source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/search/Switchboard.java |
Sat Apr 22 23:32:40 CEST 2017 by reger | upd to jwat-1.0.5 Changed Files: .classpath, build.xml, lib/jwat-archive-common-1.0.5.jar, lib/jwat-common-1.0.5.jar, lib/jwat-gzip-1.0.5.jar, lib/jwat-warc-1.0.5.jar, pom.xml |
Thu Apr 20 00:47:52 CEST 2017 by reger | fix unit test MultiProtocolURL(file) assertion for Windows path with drive letter. Changed Files: test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Thu Apr 20 00:18:18 CEST 2017 by reger | Take out mailto collect in internal parsed document As earlier plans to make use of mailto as separate webgraph entity didn't materialize (see http://forum.yacy-websuche.de/viewtopic.php?f=8&t=5726&p=32493&hilit=mailto#p32493) free the unused handling and resources. Changed Files: htroot/ViewFile.java, source/net/yacy/document/Document.java |
Sun Apr 16 04:25:29 CEST 2017 by reger | Add url input field as source for WarcImporter allowing to import warc from url without prior download. Changed Files: htroot/IndexImportWarc_p.html, htroot/IndexImportWarc_p.java, source/net/yacy/document/importer/WarcImporter.java |
Fri Apr 14 14:23:50 CEST 2017 by luccioman | Improved http client close time on stream processing errors. Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java |
Wed Apr 12 17:17:03 CEST 2017 by luccioman | Fixed endless loop case in wikicode processing. Detected when importing recent MediaWiki dumps containing some pages with script content in plain text format (see Scribunto extension https://www.mediawiki.org/wiki/Extension:Scribunto ). Further improvement : modify the MediawikiImporter to prevent processing revisions whose <model> is not wikitext. Changed Files: source/net/yacy/data/wiki/WikiCode.java, test/java/net/yacy/data/wiki/WikiCodeTest.java |
Wed Apr 12 09:23:10 CEST 2017 by luccioman | Improved support for non ASCII chars in local file system URLs Creating a MultiProtocolURL instance from a File object and then retrieving a File with getFSFile() was inconsistent with file paths containing space or non ASCII chars. Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Tue Apr 11 08:21:34 CEST 2017 by luccioman | Improved error reports on various wiki dump prerequisites failure cases. Also added some JavaDoc. Changed Files: htroot/IndexImportMediawiki_p.html, htroot/IndexImportMediawiki_p.java |
Tue Apr 11 07:34:17 CEST 2017 by luccioman | Used a text input for wiki dump import file selection. Using an HTML "file" input was confusing (as reported by promocore on YaCy forum : http://forum.yacy-websuche.de/viewtopic.php?f=5&t=5965) , and it only worked with MS IE/Edge on a local YaCy peer : - for security reasons some current major browsers such as Firefox or Chrome do not allow to send full file path information when using a file form input - the local file system selection popup doesn't make sense when you want to import a dump on a remote YaCy server Changed Files: htroot/IndexImportMediawiki_p.html |
Mon Apr 10 22:58:20 CEST 2017 by reger | Adjust ConfigSearchPage_p to activated hosts navigator as plugin Changed Files: htroot/ConfigSearchPage_p.html, htroot/ConfigSearchPage_p.java |
Mon Apr 10 22:42:06 CEST 2017 by reger | Activate hosts navigator plugin. This includes rwi results in the navigator count. This might be tangential related to http://mantis.tokeek.de/view.php?id=736 as the example includes a local index search, while rwi results are not counted. Changed Files: htroot/yacysearchtrailer.html, htroot/yacysearchtrailer.java, htroot/yacysearchtrailer.json, htroot/yacysearchtrailer.xml, source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/query/QueryModifier.java, source/net/yacy/search/query/SearchEvent.java |
Sun Apr 09 21:42:05 CEST 2017 by reger | add missing text from ConfigRobotsTxt_p to master.lng and link to Translation Editor to Translation News page. Changed Files: htroot/TransNews_p.html, locales/master.lng.xlf |
Sun Apr 09 02:09:32 CEST 2017 by reger | add servlet to list user in UserDB and made user editor available in separate servlet for a quick and easy overview of configured user and selection for edit. Changed Files: htroot/ConfigAccountList_p.html, htroot/ConfigAccountList_p.java, htroot/ConfigAccounts_p.html, htroot/ConfigAccounts_p.java, htroot/ConfigUser_p.html, htroot/ConfigUser_p.java |
Sat Apr 08 22:54:57 CEST 2017 by reger | fix edit current user form to required post mehtod introduced with https://github.com/yacy/yacy_search_server/commit/cde237b68763c542da20038e5f62bea341ae1d37 Changed Files: htroot/ConfigAccounts_p.html, htroot/ConfigAccounts_p.java |
Fri Apr 07 09:15:05 CEST 2017 by Michael Peter Christen | added flatjson parser (stub, unfinished) Changed Files: source/net/yacy/search/Switchboard.java |
Wed Apr 05 00:08:25 CEST 2017 by reger | Introduce a Keyword search navigator using the index field keywords. The keywords field string is split into words as navigator entries. A keyword navigator facet is essential for search appliance usage were documents and metadata use often specialized keyword vocabularies to filter search results. This navi can be used without custom index schema. As we don't have defined a search query command to filter "keywords" yet, the filtering is limited by adding the keyword to the search query. Changed Files: source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/navigator/TokenizedStringNavigator.java |
Mon Apr 03 22:53:07 CEST 2017 by reger | add CookieTest_p.html text to master.lng Changed Files: locales/master.lng.xlf |
Mon Apr 03 12:20:16 CEST 2017 by luccioman | Enforced access controls on a few more administration pages. - ensure use of HTTP POST method when performing server side effect operations - transaction token required to ensure the request has effectively been requested by user interaction Changed Files: htroot/ConfigPortal_p.html, htroot/ConfigPortal_p.java, htroot/Table_API_p.html, htroot/Table_API_p.java, htroot/Translator_p.html, htroot/Translator_p.java |
Mon Apr 03 11:40:37 CEST 2017 by luccioman | Escaped HTML eventually active content from recorded API call comments. Changed Files: htroot/Table_API_p.java |
Sun Apr 02 22:30:23 CEST 2017 by reger | update master.lng with recent text changes to IndexExport_p.html, IndexImportWarc_p.html Changed Files: locales/master.lng.xlf |
Sun Apr 02 20:36:22 CEST 2017 by reger | use css error class for error msg in IndexImportOAIPMH_p.html, adjust to xhtml <p> usage rule Changed Files: htroot/IndexImportOAIPMH_p.html |
Sun Apr 02 03:59:37 CEST 2017 by reger | remove test case for Standard_MemoryControl which will always fail see https://github.com/yacy/yacy_search_server/pull/114 Changed Files: |
Sun Apr 02 03:32:21 CEST 2017 by reger | Add servlet to import warc file from filesystem IndexImportWarc_p.html. Apply Importer interface to WarcImporter Changed Files: htroot/IndexImportWarc_p.html, htroot/IndexImportWarc_p.java, htroot/env/templates/submenuIndexImport.template, source/net/yacy/document/importer/WarcImporter.java, source/net/yacy/search/Switchboard.java |
Sat Apr 01 01:04:17 CEST 2017 by Michael Peter Christen | added export to elasticsearch. The export dump can easily be imported to elasticsearch using the command curl -XPOST localhost:9200/collection1/yacy/_bulk --data-binary @yacy_dump_XXX.flatjson Changed Files: htroot/IndexExport_p.html, htroot/IndexExport_p.java, source/net/yacy/cora/federate/solr/responsewriter/FlatJSONResponseWriter.java, source/net/yacy/search/index/Fulltext.java |
Thu Mar 30 16:14:22 CEST 2017 by luccioman | URL Viewer : only display the link to metadata when metadata exists Changed Files: htroot/ViewFile.html, htroot/ViewFile.java |
Thu Mar 30 10:23:47 CEST 2017 by luccioman | Modified RWI settings page radio click event to use HTTP POST Changed Files: htroot/IndexControlRWIs_p.html, locales/de.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Thu Mar 30 09:22:28 CEST 2017 by luccioman | Updated API calls recording/replay with recent changes. - enabled HTTP POST calls with Digest HTTP authentication - made API calls compatible with API newly restricted to HTTP POST only with transaction token validation - ensured backward compatibility with older entries recorded as HTTP GET Changed Files: htroot/CrawlStartScanner_p.java, source/net/yacy/data/WorkTables.java |
Sun Mar 26 23:52:31 CEST 2017 by reger | fix default/httpd.mime Z file extension to lower case + test case Changed Files: defaults/httpd.mime, test/java/net/yacy/cora/document/analysis/ClassificationTest.java |
Sun Mar 26 23:26:40 CEST 2017 by reger | remove seedlist bootstrap target (not working for some longer time) Changed Files: defaults/yacy.network.freeworld.unit |
Sun Mar 26 23:13:12 CEST 2017 by reger | Add label text for search word statistic (AccessTracker_p.html) to master lng file Changed Files: locales/master.lng.xlf |
Sun Mar 26 20:05:48 CEST 2017 by reger | One more use of SwitchboardConstants.SERVER_PORT constant, apply standard servlet design pattern initialization of solrselectservlet Changed Files: source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java |
Sun Mar 26 11:29:04 CEST 2017 by luccioman | Extended Apache HTTP Digest Auth. for use of YaCy encoded password When programmatically requesting the local peer with Apache http client, authentication credentials must be passed as clear-text values. This extension to the apache org.apache.http.impl.auth.DigestScheme permits use of the YaCy encoded password stored in the adminAccountBase64MD5 configuration property. Changed Files: source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/protocol/http/auth/HttpEntityDigester.java, source/net/yacy/cora/protocol/http/auth/YaCyDigestScheme.java, source/net/yacy/cora/protocol/http/auth/YaCyDigestSchemeFactory.java |
Sun Mar 26 10:59:04 CEST 2017 by luccioman | Updated dump/restore shell scripts : the API is now IndexExport_p.html Changed Files: bin/indexdump.sh, bin/indexrestore.sh |
Tue Mar 21 01:16:16 CET 2017 by reger | Update master lng file with added text in Settings_ServerAccess remove outdated file entry in fr.lng & sk.lng Changed Files: README.md, locales/fr.lng, locales/master.lng.xlf, locales/sk.lng |
Tue Oct 25 05:06:42 CEST 2016 by Karl-Philipp Richter | adjusted .travis.yml to build in libbuild first (see http://mantis.tokeek.de/view.php?id=545); added test of build instructions Changed Files: .travis.yml |
Mon Mar 20 02:33:21 CET 2017 by reger | Add hint how to build with maven (for the first time) to readme Changed Files: README.md |
Sun Mar 19 21:45:33 CET 2017 by reger | Add hint text to default ServerAcess Port Settings page Changed Files: htroot/Settings_ServerAccess.inc |
Sun Mar 19 07:12:35 CET 2017 by reger | Display the local search word statistic in alphabetic order Changed Files: htroot/AccessTracker_p.java, source/net/yacy/cora/sorting/OrderedScoreMap.java |
Sat Mar 18 20:32:53 CET 2017 by reger | upd to slf4j-1.7.24.jar Changed Files: .classpath, build.xml, lib/jcl-over-slf4j-1.7.24.jar, lib/log4j-over-slf4j-1.7.24.jar, lib/slf4j-api-1.7.24.jar, lib/slf4j-jdk14-1.7.24.jar, pom.xml |
Sat Mar 18 20:06:58 CET 2017 by reger | upd to icu4j-58_2.jar Changed Files: .classpath, build.xml, lib/icu4j-58_2.jar, pom.xml |
Fri Mar 17 02:19:33 CET 2017 by reger | update to jsoup-1.10.2.jar Changed Files: .classpath, build.xml, lib/jsoup-1.10.2.jar, pom.xml |
Fri Mar 17 02:07:02 CET 2017 by reger | update to jsch-0.1.54.jar Changed Files: .classpath, build.xml, lib/jsch-0.1.54.License, lib/jsch-0.1.54.jar, pom.xml |
Wed Mar 15 22:36:53 CET 2017 by reger | update translation for ConfigNetwork_p.html Changed Files: htroot/ConfigNetwork_p.html, locales/de.lng, locales/master.lng.xlf |
Wed Mar 15 01:39:15 CET 2017 by reger | make digest default authentication in defaults/web.xml Changed Files: defaults/web.xml |
Mon Mar 13 03:08:44 CET 2017 by reger | remove double occuance of geo:lat in rss tokens Changed Files: source/net/yacy/cora/document/feed/RSSMessage.java |
Mon Mar 13 00:34:40 CET 2017 by reger | upd to metadata-extractor-2.10.1.jar Changed Files: .classpath, build.xml, lib/metadata-extractor-2.10.1.License, lib/metadata-extractor-2.10.1.jar, pom.xml |
Sun Mar 12 01:54:56 CET 2017 by reger | implement RequestHeader getRequestURI, getRequestURL for legacy request Changed Files: source/net/yacy/cora/protocol/RequestHeader.java |
Thu Mar 09 22:57:51 CET 2017 by reger | remove unused import pdfParser Changed Files: source/net/yacy/document/parser/pdfParser.java |
Thu Mar 09 22:56:33 CET 2017 by reger | Improve pdf text extraction resource handling. For sort pdf <= 3 pages use already extracted content, only for long pdf > 3 pages reassign content and close internal writer (to direct free buffers) Changed Files: source/net/yacy/document/parser/pdfParser.java |
Thu Mar 09 22:50:19 CET 2017 by reger | upd to pdfbox-2.0.4.jar Changed Files: .classpath, build.xml, lib/fontbox-2.0.4.License, lib/fontbox-2.0.4.jar, lib/pdfbox-2.0.4.License, lib/pdfbox-2.0.4.jar, pom.xml |
Thu Mar 09 01:42:36 CET 2017 by reger | eliminate some compiler unchecked and deprecation warnings in nav plugins by explicite type declaration and replacing date.getYear with Calendar.get Changed Files: source/net/yacy/search/navigator/NavigatorPlugins.java, source/net/yacy/search/navigator/YearNavigator.java |
Wed Mar 08 22:35:48 CET 2017 by reger | upd to httpclient v4.5.3 Changed Files: .classpath, build.xml, lib/httpclient-4.5.3.jar, lib/httpcore-4.4.6.License, lib/httpcore-4.4.6.jar, lib/httpmime-4.5.3.jar, pom.xml |
Wed Mar 08 10:27:18 CET 2017 by luccioman | Fixed unresolved pattern case in search results progress bar. This is a fix for mantis 715 (http://mantis.tokeek.de/view.php?id=715). A possible path scenario that could leading to this case : - YaCy is running low in memory - a search is requested - before the end of search results rendering, the cleanup job runs and deletes the running search event from the cache because of short memory - then yacysearchitem renders with "-UNRESOLVED_PATTERN-" parameter values passed to the statistics() JavaScript function Changed Files: htroot/yacysearchitem.html, htroot/yacysearchitem.java |
Sun Mar 05 02:26:10 CET 2017 by reger | Extend DCEntry.getLanguage convert to ISO639-1 codes for more languages by using icu.ULocale for languages not already covered (ICU normalizes to ISO639-1 2 char codes). Add test class Use DublinCore vocabulary declarations in DCEntry and SurrogateReader for easier usage debugging, Init SurrogateReader.inputSource on first use. Changed Files: source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/SurrogateReader.java, test/java/net/yacy/document/content/DCEntryTest.java |
Sat Mar 04 22:45:17 CET 2017 by reger | further avoid to set connect info properties as header value following comment "use of properties as header values is discouraged" in case where (proxy)HTTPClient overwrites values with supplied url. Use defined request.referer procedure in response class. Changed Files: source/net/yacy/crawler/retrieval/Response.java, source/net/yacy/http/servlets/UrlProxyServlet.java, source/net/yacy/http/servlets/YaCyProxyServlet.java, source/net/yacy/server/http/HTTPDProxyHandler.java |
Sat Mar 04 19:41:31 CET 2017 by reger | use pre-defined "Connection" header key, replace depreceated Changed Files: source/net/yacy/cora/federate/solr/instance/RemoteInstance.java, source/net/yacy/cora/protocol/http/HTTPClient.java |
Fri Mar 03 12:05:30 CET 2017 by luccioman | Added an advanced settings page for referrer policy settings. Feedback will be welcome, notably on the descriptive content of this page. Changed Files: htroot/SettingsAck_p.html, htroot/SettingsAck_p.java, htroot/Settings_Referrer.inc, htroot/Settings_p.html, htroot/Settings_p.java, source/net/yacy/http/ReferrerPolicy.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, source/net/yacy/search/SwitchboardConstants.java |
Fri Mar 03 00:21:56 CET 2017 by reger | fix proxyservlet response url to respect http scheme if a relative Location header is returned. Changed Files: source/net/yacy/http/servlets/UrlProxyServlet.java, source/net/yacy/http/servlets/YaCyProxyServlet.java |
Wed Mar 01 09:43:00 CET 2017 by luccioman | Updated Archive-It heuristics URL. The archive-it OpenSearch URL requested without restriction on collections ("i" parameter) almost always ends up with timeout or fails. Changed Files: defaults/heuristicopensearch.conf |
Mon Feb 27 23:00:46 CET 2017 by reger | fixed ReindexSolrBusyThread new and unexpected repeat of same query with low number of found documents - by adding additional end condition to remove processed query with number of found docs <= process-chunck-size. Noticed on query h4_txt:[* TO *], found 21, process 21, call of commit happend but on next cycle same query again 21 docs found (while h4_txt was removed from schema and committed inputdocuments). Changed Files: source/net/yacy/search/index/ReindexSolrBusyThread.java |
Mon Feb 27 01:04:31 CET 2017 by reger | fix delta time calculation in PerformanceSearch_p for the 1. entry (INITIALIZATION displayed absolute date, set delta to 0 for 1. entry) Changed Files: htroot/PerformanceSearch_p.java |
Sun Feb 26 11:03:15 CET 2017 by luccioman | Fixed datacite.org heuristics base url. The datacite Solr search http URL was returning http status 301 in order to redirect to its https version, thus making that YaCy heuristic always fail. Changed Files: defaults/federatecfg/datacite.solr.schema |
Sun Feb 26 02:39:52 CET 2017 by reger | Adjust DefaultServlet test case to recent change, depreciate unused CONNECTION_PROP_PROTOCOL (also as it might be misleading with getProtocol vs getScheme) Changed Files: source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, test/java/net/yacy/http/servlets/YaCyDefaultServletTest.java |
Sat Feb 25 23:55:17 CET 2017 by reger | Fix call parameter for ConnectionInfo in MonitorHandler (expected scheme e.g. http, was protocol version). Depreceate obsolete custom X-...-Scheme header constant. Use existing FORMAT_ANSIC Dateformatter in HeaderFramework. Correct htmlParserTest (del one not intended println) Changed Files: source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/http/MonitorHandler.java, source/net/yacy/http/servlets/YaCyDefaultServlet.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Fri Feb 24 11:09:42 CET 2017 by luccioman | Added a hint title for required fields in the Solr Schema editor Changed Files: htroot/IndexSchema_p.html |
Fri Feb 24 11:08:18 CET 2017 by luccioman | Switched a few more Solr fields from strictly mandatory to optional Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionSchema.java |
Fri Feb 24 01:25:32 CET 2017 by reger | fix htmlParser <script> text extraction on code containing expression recognized as tag like 1<a reported in https://github.com/yacy/yacy_search_server/issues/109 Script content is ignored by default, but the text is filtered for html tags. Modified scraper to skip tag filtering while within a <script> section (until a closing tag is detected </script>. Possible side effect, missing </script> end-tag will truncate trailing content text. Changed Files: source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/TransformerWriter.java, test/java/net/yacy/document/parser/htmlParserTest.java |
Thu Feb 23 11:09:43 CET 2017 by luccioman | Improved MultiprocotolURL non ASCII characters support. After @sinkuu Pull Request #108 added JUnit tests, updated some JavaDoc and also improved URL tokenization to support non ASCII characters. Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java, test/java/net/yacy/cora/document/id/MultiProtocolURLTest.java |
Thu Feb 23 07:52:55 CET 2017 by luccioman | Merge pull request #110 from goofy-bz/patch-1 Fixing some typos Changed Files: locales/fr.lng |
Thu Feb 23 01:13:31 CET 2017 by goofy-bz | Fixing some typos up to line #1000 only Changed Files: locales/fr.lng |
Thu Feb 23 00:27:56 CET 2017 by reger | Correct dublincore title property text to lowercase in htmlresponsewriter, remove unused (carry over) local variable Do the same for other responsewriter. Changed Files: source/net/yacy/cora/federate/solr/responsewriter/EnhancedXMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/HTMLResponseWriter.java, source/net/yacy/cora/federate/solr/responsewriter/OpensearchResponseWriter.java |
Wed Feb 22 02:01:48 CET 2017 by Burkhard | Update SearchEvent.java Fix NPE on disabled local SolrIndex, occuring on search moving to the 2nd result page. The debug purpose only setting to disabeling local SolrIndex (System Admin -> Debug Settings) should long term probably be removed from production code. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Feb 21 22:59:11 CET 2017 by luccioman | Switched some Solr fields from mandatory to optional These fields are default enabled but with no doubt not strictly mandatory with the current code base. As reported by @reger24, splitting between essential mandatory and optional fields is still to be improved to reflect the current YaCy needs. Changed Files: defaults/solr.collection.schema, source/net/yacy/search/schema/CollectionSchema.java |
Mon Feb 20 23:27:33 CET 2017 by reger | Add extract of queries.log in form of top search word cloud (last 7 days) to AccessTracker_p.html (Network Access -> Local Search Log page). It displays top 20 words of search queries. Changed Files: htroot/AccessTracker_p.html, htroot/AccessTracker_p.java |
Mon Feb 20 00:14:14 CET 2017 by reger | correct fromDate init value on missing param in api/timeline_p servlet revert test modification from last commit in AccessTracker.main Changed Files: htroot/api/timeline_p.java, source/net/yacy/search/query/AccessTracker.java |
Sun Feb 19 05:23:17 CET 2017 by reger | add hint of query syntax in AccessTracker log (qs=normal querystring, sq=solr-querystring) to allow to filter simple text queries for processing, remove toString for counter parameter use more predefined constants in solrservlet Changed Files: source/net/yacy/http/servlets/GSAsearchServlet.java, source/net/yacy/http/servlets/SolrSelectServlet.java, source/net/yacy/search/query/AccessTracker.java |
Fri Feb 17 11:09:30 CET 2017 by luccioman | Fixed a NullPointerException case possible on Index Export As reported by Palulukas in YaCy forum (http://forum.yacy-websuche.de/viewtopic.php?f=18&t=5944&sid=dcef5b899ab4aa9b40e3a3d158c13aed#p33454) the Index Export operation can fails, notably when the Solr index contains one or more documents with empty (despite required) "load_date_dt" field. This fixes the export failure when the situation finally occurs, but more should be done to harden verifications on minimum required fields. Changed Files: source/net/yacy/search/index/Fulltext.java |
Thu Feb 16 01:43:14 CET 2017 by reger | Reduce self generated content for text_t (visible text index field) to avoid repeat of tokenized url as description, continuation of https://github.com/yacy/yacy_search_server/commit/7e09bff4a1a117d2f2336e004ec67ffb325a7e9d https://github.com/yacy/yacy_search_server/commit/1409cabe8b7bce1fb767f01665d9d7e0a91a81b6 Add some javadoc, and not needed remove of omitted fields in postprocessing. Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Wed Feb 15 23:26:54 CET 2017 by reger | removed faroo news from default opensearch config As @luccioman informed, it's only useable with a free api key http://www.faroo.com/hp/api/api.html http://blog.faroo.com/2013/06/30/faroo-introduces-an-api-key/ Changed Files: defaults/heuristicopensearch.conf |
Wed Feb 15 15:04:40 CET 2017 by luccioman | Added robots.txt support for heuristics federated search. As noticed by @reger24, abusive use of OpenSearch systems should be prevented, especially if allowing to parse and reuse HTML results. robots.txt file is now checked before requesting an external OpenSearch system to respect the host exclusions and eventual crawl-delay value. The check is also performed when trying to add a new OpenSearch URL template through the /ConfigHeuristics_p.html admin page. Changed Files: htroot/ConfigHeuristics_p.java, source/net/yacy/cora/federate/FederateSearchManager.java |
Sat Feb 11 08:10:14 CET 2017 by sinkuu | Use java.net.URLDecoder Changed Files: source/net/yacy/cora/document/id/MultiProtocolURL.java |
Tue Feb 14 02:30:26 CET 2017 by reger | adjust translation to renamed configparser_p.html Changed Files: locales/cn.lng, locales/de.lng, locales/hi.lng, locales/ja.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Tue Feb 14 02:04:42 CET 2017 by reger | make ConfigParser a protected page, for consistent behavior of locked menu items. Changed Files: htroot/ConfigParser_p.html, htroot/ConfigParser_p.java, htroot/env/templates/submenuCrawler.template |
Tue Feb 14 00:31:32 CET 2017 by reger | update opensearch conf - remove suche.sueddeutsche.de apparently they've revoked the participation in opensearch initiative. Changed Files: defaults/heuristicopensearch.conf |
Fri Feb 10 09:40:42 CET 2017 by luccioman | Upgraded Apache Ant to 1.10.1 in the Docker alpine flavor image For a more reliable Docker image build, also switched to the ant archive repository to fetch the needed binary as other repositories only provide the latest versions. Changed Files: docker/Dockerfile.alpine |
Thu Feb 09 16:42:21 CET 2017 by luccioman | Replaced absolute redirection locations by relative ones when possible. This makes integration of YaCy behind a reverse proxy subfolder easier. Changed Files: htroot/Blacklist_p.java, htroot/Status.java, htroot/Wiki.java, source/net/yacy/repository/BlacklistHelper.java |
Mon Feb 06 12:41:24 CET 2017 by luccioman | Improved termination of timed out remote solr requests to peers. On timeout, closing remote Solr requests is proper than simply using Thread.interrupt() that is not effective in most cases. Closing does not ask commit on remote solr, but release http connections resources and is more likely to end those threads that can else wait indefinitely. Other related improvements included : - no more marking remote peer as not available when remote search is interrupted before timeout by the cleanup job. - added a short fine log level trace of failing remote solr requests Changed Files: source/net/yacy/peers/Protocol.java |
Fri Feb 03 10:32:31 CET 2017 by luccioman | Removed deprecated "localMissCount" prop from yacysearchlatestinfo.json. This property has been deprecated four years ago by commit d74472f5625ff097e7541e1a56156cbe487b2651. For any active search event id, it was then always filled with "-UNRESOLVED_PATTERN-". Changed Files: htroot/yacysearchlatestinfo.java, htroot/yacysearchlatestinfo.json |
Fri Feb 03 09:55:08 CET 2017 by luccioman | Named a Thread without name for easier monitoring Changed Files: source/net/yacy/search/query/SearchEvent.java |
Fri Feb 03 09:54:29 CET 2017 by luccioman | Distinguished solr connectors thread names for easier monitoring. Changed Files: source/net/yacy/cora/federate/solr/connector/EmbeddedSolrConnector.java, source/net/yacy/cora/federate/solr/connector/RemoteSolrConnector.java |
Wed Feb 01 18:44:42 CET 2017 by luccioman | Refactored the DHT-Trigger section in Performance_p.html page. This is to be more easily understandable and to reflect more accurately the current memory strategies implementations that eventually set the "proper" state not only because DHT reception. Changed Files: htroot/Performance_p.html, locales/cn.lng, locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng, locales/uk.lng |
Tue Jan 31 16:33:17 CET 2017 by luccioman | Updated French translation for the /Performance_p.html page. Also updated the master xliff file with missing recent changes. Changed Files: locales/fr.lng, locales/master.lng.xlf |
Tue Jan 31 09:20:19 CET 2017 by luccioman | Fixed unresolved pattern on directory entries in HostBrowser.xml api. As described in mantis 725 (http://mantis.tokeek.de/view.php?id=725) the HostBrowser.xml api directory entries had incorrect count attribute value. This was because the HostBrowser html page and backing template servlet evolved, but modifications were not reported on the xml api. Changed Files: htroot/HostBrowser.xml |
Mon Jan 30 22:44:28 CET 2017 by reger | adjust column layout in Settings_Proxy.inc Changed Files: htroot/Settings_Proxy.inc |
Sat Jan 28 10:19:39 CET 2017 by luccioman | Added a CSS class for infobox block. This will prevent mistakenly hiding a div element not designed to be an infobox but having a ".info" parent (After having previously added the possibility for a div - and not only a span element - to be an infobox). Changed Files: htroot/Performance_p.html, htroot/env/base.css |
Sat Jan 28 01:13:57 CET 2017 by reger | Update language file de & master, remove obsolete "Augmented Browsing" Changed Files: locales/de.lng, locales/master.lng.xlf |
Sat Jan 28 00:36:03 CET 2017 by reger | Add consistency check for related index fields upon load and save of index schema. To assemble the original link url for out-/inboundlinks, icons and pictures the *_protocol_sxt and *_urlstub_sxt is needed (due to the used data-reduced storage methode). Auto-enable *_protocol_sxt if *_urlstub_sxt is enabled. to be able to correctly assemble the original link url. Changed Files: source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Jan 26 23:49:15 CET 2017 by reger | adjust the Field-Reindex Thread to verify and update the document id in case hash (ID) doesn't match document url (sku field). Changed Files: source/net/yacy/search/index/ReindexSolrBusyThread.java |
Thu Jan 26 06:37:29 CET 2017 by Michael Christen | Merge pull request #98 from Velociraptor85/patch-2 LSB Tag Changed Files: addon/yacyInit.sh |
Thu Jan 26 06:29:42 CET 2017 by Michael Christen | Merge pull request #105 from ivar/patch-1 Update README.md - removes deprecated URL Changed Files: README.md |
Thu Jan 26 05:36:48 CET 2017 by Ivar Vasara | Update README.md - removes deprecated URL Changed Files: README.md |
Thu Jan 26 01:13:32 CET 2017 by luccioman | Improved Index Browser accessibility with semantically richer html tags. Made use of ol, li, thead, th, tbody, h1 and h2 html tags. Added aria-label attributes to provide alternative textual information previously only conveyed by color cue. Tested behavior with NVDA 2016.4 screen reader. Changed Files: htroot/HostBrowser.html |
Wed Jan 25 09:54:39 CET 2017 by luccioman | Fixed local image search pagination regression. As reported by @tglman on issue #90, when searching images on the local index only, pages next to the first were always empty. This was a regression from commit c25e48e969f180dcc3c73863acbfcc383a182c8f. Changed Files: source/net/yacy/search/query/SearchEvent.java |
Tue Jan 24 17:14:14 CET 2017 by luccioman | Updated master xliff file with missing entries for HostBrowser.html. Also translated lang="en" html attribute to lang="[targetLang]" on locale files having translated entries for HostBrowser.html Changed Files: locales/de.lng, locales/fr.lng, locales/master.lng.xlf, locales/ru.lng |
Tue Jan 24 15:56:29 CET 2017 by Michael Peter Christen | added dc.date.modified and dc.date.created to date parser Changed Files: source/net/yacy/document/parser/html/ContentScraper.java |
Tue Jan 24 11:38:56 CET 2017 by luccioman | Updated French translation of HostBrowser.html Changed Files: locales/fr.lng |
Tue Jan 24 09:40:43 CET 2017 by luccioman | Fixed Index Browser page HTML validation errors and switched to HTML5. Also removed deprecated HTML attributes uses. Validation performed with Nu Html Checker 17.1.0. Cross browser tested with : - Debian Jessie : Firefox ESR 45.6.0 - MS Windows 10 : Firefox 50.1.0, Chrome 55.0.2883.87, MS Edge Changed Files: htroot/HostBrowser.html, htroot/HostBrowser.java, htroot/HostBrowserAdmin_p.html |
Tue Jan 24 01:51:28 CET 2017 by reger | assure that RWI Index.Segment IODispatcher is not blocking on shudown waiting on a semaphore permit. see desc. http://mantis.tokeek.de/view.php?id=723 Changed Files: source/net/yacy/kelondro/rwi/IODispatcher.java |
Mon Jan 23 16:05:51 CET 2017 by luccioman | Documented /HostBrowser.html related configuration settings Changed Files: defaults/yacy.init, htroot/HostBrowser.java |
Mon Jan 23 14:49:02 CET 2017 by luccioman | Display Index Browser links requiring auth only when authenticated. In the /HostBrowser.html page "only hosts with urls pending in the crawler", "only with load errors" and "Administration Options" all require administration credentials. But they were displayed even to unauthenticated users, and clicking them did nothing and returned the /HostBrowser.html page empty. Changed Files: htroot/HostBrowser.html, htroot/HostBrowser.java |
Sun Jan 22 12:31:14 CET 2017 by luccioman | Fixed display of crawler pending URLs counts in HostBrowser.html page. As described in mantis 722 (http://mantis.tokeek.de/view.php?id=722) Also updated some Javadoc. Changed Files: htroot/HostBrowser.java, source/net/yacy/crawler/Balancer.java, source/net/yacy/crawler/HostBalancer.java, source/net/yacy/crawler/data/NoticedURL.java |
Sun Jan 22 12:19:43 CET 2017 by luccioman | Removed temporary test main method commited by mistake. Changed Files: htroot/yacysearch.java |
Sun Jan 22 00:01:18 CET 2017 by reger | add ukr and pol to DCEntry.getLanguage ISO639-2 3-char language code conversion to deliver uk, pl 2-char code and use if else to return on match Changed Files: source/net/yacy/document/content/DCEntry.java |
Sat Jan 21 01:53:43 CET 2017 by reger | delete outdated and unmaintained Netbeans project Netbeans has good build-in maven support which is a supported and maintained build env, making special and additional NB setting obsolete. Changed Files: |
Fri Jan 20 02:15:11 CET 2017 by reger | upd to commons-compress-1.13.jar hide external icon on forge logo (was also out of position in IE) Changed Files: .classpath, build.xml, htroot/Status.html, lib/commons-compress-1.13.License, lib/commons-compress-1.13.jar, pom.xml |
Thu Jan 19 12:30:44 CET 2017 by luccioman | Added an optional parameter to webstructure.xml api. This new "documentStructure" parameter can be set to false to only get hosts accumulated references on a resource and thus prevent scraping the specified URL and getting citations references. Also set WebStructureGraph constants as final and updated the Javadoc with example api call URLs. Changed Files: htroot/api/webstructure.java, source/net/yacy/peers/graphics/WebStructureGraph.java |
Tue Jan 17 23:45:56 CET 2017 by reger | remove obsolete lastmodified calculation in WebgraphConfig Changed Files: source/net/yacy/search/schema/WebgraphConfiguration.java |
Tue Jan 17 17:01:56 CET 2017 by luccioman | Updated Javadoc and Junit tests for the WebStructureGraph class. Changed Files: source/net/yacy/peers/graphics/WebStructureGraph.java, test/java/net/yacy/peers/graphics/WebStructureGraphTest.java |
Tue Jan 17 15:59:55 CET 2017 by luccioman | Made sure webstructure.xml API produces valid XML. Host names should not contain XML special characters such as quotation mark, but at this stage the WebGraph may have mistakenly recorded a host name with such characters. What's more the DigestURL constructor does not prevent this. By the way using serverObjects.putXML to encode host names we ensure here the rendered XML is well formed and can be parsed by external tools even if an structure entry is incorrect. Changed Files: htroot/api/webstructure.java |
Mon Jan 16 18:41:58 CET 2017 by luccioman | Fixed WatchWebStructure_p.html render to include https URLs. As described in mantis 721 (http://mantis.tokeek.de/view.php?id=721) WatchWebStructure_p.html failed to include in its structure view https and other protocols and ports than default http. Changed Files: htroot/WebStructurePicture_p.java, source/net/yacy/peers/graphics/WebStructureGraph.java, test/java/net/yacy/peers/graphics/WebStructureGraphTest.java |
Mon Jan 16 16:41:06 CET 2017 by luccioman | Fixed webstructure.xml API used with a domain name 'about' parameter. As described in mantis 720 (http://mantis.tokeek.de/view.php?id=720), when requesting this API with a domain name instead of a complete URL only HTTP references on default port were listed. Changed Files: htroot/api/webstructure.java, source/net/yacy/peers/graphics/WebStructureGraph.java, test/java/net/yacy/peers/graphics/WebStructureGraphTest.java |
Mon Jan 16 10:18:42 CET 2017 by luccioman | Factored code re-implementing DigestURL.hosthash() method. This ensure consistent implementation of the url host hash generation and easier usage finding in source code. Also added a unit test for this function. Changed Files: htroot/WebStructurePicture_p.java, source/net/yacy/cora/document/id/DigestURL.java, source/net/yacy/crawler/CrawlStacker.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/search/Switchboard.java, test/java/net/yacy/cora/document/id/DigestURLTest.java |
Fri Jan 13 16:10:59 CET 2017 by luccioman | Added automated unit tests and perfs test for WebStructureGraph class. Fixed references count when multiple links target the same domain name in one document. Changed Files: source/net/yacy/peers/graphics/WebStructureGraph.java, test/java/net/yacy/peers/graphics/WebStructureGraphTest.java |
Fri Jan 13 16:05:46 CET 2017 by luccioman | Factored common code with DigestURL.hosthash() Changed Files: htroot/HostBrowser.java, htroot/api/webstructure.java |
Thu Jan 12 17:52:47 CET 2017 by luccioman | Detailed some Javadoc related to /api/webstructure.xml usage. Changed Files: htroot/api/webstructure.java, source/net/yacy/peers/graphics/WebStructureGraph.java |
Thu Jan 12 01:36:30 CET 2017 by reger | Start to rename "Augmented Browsing" to "Web Proxy ..." / "View via Proxy" The augmented Browsing option was reduced to the web proxy functionallity. Augmented browsing is not available and no known plan exist to reimplement alteration of result pages with additional information. Changed Files: htroot/AugmentedBrowsing_p.html, htroot/ConfigSearchPage_p.html, htroot/yacysearchitem.html, locales/de.lng, locales/master.lng.xlf |
Mon Jan 09 16:45:31 CET 2017 by luccioman | Ignore generated Javadoc with git SCM. Changed Files: .gitignore |
Sat Jan 07 18:24:29 CET 2017 by reger | fix DC.Elements namespace in DublinCore vocabulary class delete redundant (unused) DCElements. Changed Files: source/net/yacy/cora/lod/vocabulary/DublinCore.java |
Fri Jan 06 12:24:31 CET 2017 by luccioman | Blacklist import and update performance improvements. Measurement sample : import from blacklist local file containing about 15000 entries - before refactoring : several minutes - after refactoring : a few seconds! Changed Files: htroot/BlacklistCleaner_p.java, htroot/IndexControlRWIs_p.java, htroot/sharedBlacklist_p.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/BlacklistHostAndPath.java |
Fri Jan 06 11:23:40 CET 2017 by luccioman | Added some JavaDoc. Changed Files: htroot/sharedBlacklist_p.java, source/net/yacy/server/serverObjects.java |
Fri Jan 06 09:00:28 CET 2017 by luccioman | Display result favicons only for http or https resources. Favicon display only makes sense for http(s) websites, being public or intranet. So I modified the favicon conditional display to verify the result URL protocol rather than if we are in intranet mode. Also prevented rendering an img HTML tag with empty src on other results protocols such as ftp or file. Fixing this thanks to priest2 report (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5923). Changed Files: htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchitem.json |
Fri Jan 06 03:01:52 CET 2017 by reger | fix concurrency issue with htmlParser using not current scraper data resulting in incorrect data for some html index metadata. Details see http://mantis.tokeek.de/view.php?id=717 Changed Files: source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Document.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/search/schema/CollectionConfiguration.java |
Thu Jan 05 14:54:59 CET 2017 by luccioman | Added descriptive titles to Crawler_p.html speed settings. As reported by bubul (http://forum.yacy-websuche.de/viewtopic.php?f=23&t=5924) , LF and MH acronyms meaning were not detailed. Also added label tags for improved accessibility on these input fields. Changed Files: htroot/Crawler_p.html |
Thu Jan 05 00:24:37 CET 2017 by reger | fix exception on URIMetadataNote instantiation with corrected id hash on host_id_s. Use Solr setField instead of addField to prevent java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String at net.yacy.kelondro.data.meta.URIMetadataNode.hosthash(URIMetadataNode.java:247) at net.yacy.search.query.SearchEvent.addNodes(SearchEvent.java:966) at net.yacy.peers.Protocol.solrQuery(Protocol.java:1242) at net.yacy.peers.RemoteSearch$2.run(RemoteSearch.java:349) Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java |
Mon Jan 02 14:23:25 CET 2017 by luccioman | Upgraded Apache Ant to 1.10.0 for the Alpine flavor Docker image. Changed Files: docker/Dockerfile.alpine |
Mon Jan 02 10:24:17 CET 2017 by luccioman | Adjusted crawl depth control for FTP crawl start URLs. Changed Files: source/net/yacy/crawler/CrawlStacker.java |
Mon Jan 02 03:04:21 CET 2017 by reger | Complete harmonization RequestHeader getCookie with std ServletRequest to use javax.servlet.http.Cookie parameters. Depreciate now obsolete getHeaderCookies. Adjust setting of MaxAge to spec if >= 0 otherwise keep default. Changed Files: htroot/CookieTest_p.java, htroot/User.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/data/UserDB.java, source/net/yacy/search/Switchboard.java |
Sun Jan 01 23:58:38 CET 2017 by reger | On negative result vote also delete document from fulltext index (not only from dht) Changed Files: htroot/yacysearch.java |
Sun Jan 01 23:54:18 CET 2017 by reger | Merge origin/master Changed Files: docker/Dockerfile, docker/Dockerfile.alpine, docker/Readme.md, startYACY.sh |
Sun Jan 01 23:53:44 CET 2017 by reger | fix of fulltext.remove() by id of webgraph document webgraph has document hash in source_id_s Changed Files: source/net/yacy/search/index/Fulltext.java |
Sat Dec 31 09:51:07 CET 2016 by luccioman | Fixed docker stop behavior. - Adjusted start script in debug mode to make sure the main java process can receive signals such as SIGTERM - Modified docker images main command to properly propagate SIGTERM signal to the main java process Changed Files: docker/Dockerfile, docker/Dockerfile.alpine, docker/Readme.md, startYACY.sh |
Wed Dec 28 09:47:27 CET 2016 by luccioman | Fixed YaCy proper shutdown triggered by SIGTERM signal. The main shutdown hook thread was not properly waiting for the main thread termination which consequently could not properly close resources and threads. After terminating a running YaCy peer this way (Ctrl+C in console, or kill <pid> for example), you could see the still existing DATA/yacy.running file. Tested with : - Debian Jessie openjdk 7 and 8 : regular shutdown, Ctrl+C, kill command, system restart while yacy is running - Windows 10 Oracle JDK 7 and 8 : non regression on regular shutdown Changed Files: source/net/yacy/search/Switchboard.java, source/net/yacy/yacy.java |