YaCy Release 1.1

by Michael Peter Christen

added remote search to solr on YaCy peers!
- when doing a remote search, node peers are selected for solr queries
- the solr query is done concurrently to the standard YaCy rwi search
- the solr search result is feeded into the same data structure that
prepares the rwi search result
- the same remote seach that is done to several outside peers is done to
the local solr index
- the search process works now also without any 'old' RWI data using
solr

Changed Files: .classpath, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java

by Michael Peter Christen

refactoring

Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/IndexFederated_p.java, htroot/IndexShare_p.java, htroot/QuickCrawlLink_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/YBRFetch_p.java, htroot/api/schema_p.java, htroot/api/status_p.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacy/crawlReceipt.java, htroot/yacy/query.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/ResultEntry.java

by Michael Peter Christen

code simplification

Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, htroot/api/schema_p.java, htroot/gsa/searchresult.java, htroot/solr/select.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java

by Michael Peter Christen

- refactoring (load -> getMetadata)
- added getDocument to retrieve Solr documents which shall replace
getMetadata

Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java

by Michael Peter Christen

FOR THE BRAVE.. this is a forced migration to solr which is now ready
for production as a replacement of the metadata-db.
This intermediate release 1.041 will switch on the previously optional
solr index and the old metadata-db will still work as it did before.
Solr+metadata are accessed in mixed mode, no migration is done yet.
If this causes not a catastrophe until the end of the weekend, we will
do a YaCy 1.1 main release containing this as default.

Changed Files: build.properties, defaults/yacy.init, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java

by Michael Peter Christen

get the peer location more quickly

Changed Files: htroot/Network.java, htroot/gsa/searchresult.java, htroot/solr/select.java, htroot/yacy/hello.java, htroot/yacy/query.java, htroot/yacy/transferRWI.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/dht/PeerSelection.java

by cominch

Merge remote-tracking branch 'original yacy/master'

Changed Files: .classpath, .settings/org.eclipse.jdt.core.prefs, addon/YaCy.app/Contents/Info.plist, bin/localsearch.sh, build.properties, build.xml, defaults/httpd.mime, defaults/solr.keys.list, defaults/solr/schema.xml, defaults/solr/solrconfig.xml, defaults/yacy.init, defaults/yacy.network.freeworld.unit, htroot/AccessGrid_p.java, htroot/AccessPicture_p.java, htroot/AccessTracker_p.java, htroot/AugmentedBrowsingFilters_p.java, htroot/AugmentedBrowsing_p.java, htroot/AugmentedParsing_p.java, htroot/Banner.java, htroot/BlacklistCleaner_p.java, htroot/BlacklistImpExp_p.java, htroot/BlacklistTest_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/CacheResource_p.java, htroot/ConfigAccounts_p.java, htroot/ConfigAppearance_p.java, htroot/ConfigBasic.java, htroot/ConfigHTCache_p.java, htroot/ConfigHeuristics_p.html, htroot/ConfigHeuristics_p.java, htroot/ConfigLanguage_p.java, htroot/ConfigLiveSearch.java, htroot/ConfigNetwork_p.java, htroot/ConfigPortal.java, htroot/ConfigProfile_p.java, htroot/ConfigProperties_p.java, htroot/ConfigRobotsTxt_p.java, htroot/ConfigSearchBox.java, htroot/ConfigUpdate_p.java, htroot/Connections_p.java, htroot/ContentIntegrationPHPBB3_p.java, htroot/CookieMonitorIncoming_p.java, htroot/CookieMonitorOutgoing_p.java, htroot/CrawlMonitorRemoteStart.java, htroot/CrawlProfileEditor_p.java, htroot/CrawlResults.java, htroot/CrawlStartExpert_p.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/DemoServlet.java, htroot/DemoServletInteraction.java, htroot/DemoServletRDF.java, htroot/DictionaryLoader_p.java, htroot/Help.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.html, htroot/IndexControlURLs_p.java, htroot/IndexCreateDomainCrawl_p.java, htroot/IndexCreateLoaderQueue_p.java, htroot/IndexCreateParserErrors_p.java, htroot/IndexCreateQueues_p.java, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/IndexImportMediawiki_p.java, htroot/IndexImportOAIPMHList_p.java, htroot/IndexImportOAIPMH_p.java, htroot/IndexShare_p.java, htroot/Load_MediawikiWiki.java, htroot/Load_PHPBB3.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Network.html, htroot/Network.java, htroot/NetworkPicture.java, htroot/News.java, htroot/PeerLoadPicture.java, htroot/PerformanceConcurrency_p.java, htroot/PerformanceGraph.java, htroot/PerformanceMemory_p.java, htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/PerformanceSearch_p.java, htroot/ProxyIndexingMonitor_p.java, htroot/QuickCrawlLink_p.java, htroot/Ranking_p.java, htroot/RemoteCrawl_p.java, htroot/SearchEventPicture.java, htroot/ServerScannerList.java, htroot/SettingsAck_p.java, htroot/Status.java, htroot/Supporter.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Table_RobotsTxt_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/Threaddump_p.java, htroot/Trails.java, htroot/Triple_p.java, htroot/Triplestore_p.java, htroot/User.java, htroot/ViewFile.html, htroot/ViewFile.java, htroot/ViewImage.java, htroot/ViewLog_p.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/WikiHelp.java, htroot/YBRFetch_p.java, htroot/YMarks.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/api/bookmarks/xbel/xbel.java, htroot/api/config_p.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/latency_p.java, htroot/api/schema_p.java, htroot/api/status_p.java, htroot/api/table_p.java, htroot/api/termlist_p.java, htroot/api/timeline.java, htroot/api/trail_p.java, htroot/api/version.java, htroot/api/webstructure.java, htroot/api/yacydoc.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_metadata.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/import_ymark.java, htroot/api/ymarks/manage_tags.java, htroot/api/ynetSearch.java, htroot/compare_yacy.java, htroot/env/style.java, htroot/env/templates/submenuBlacklist.template, htroot/imagetest.java, htroot/interaction/GetRDF.java, htroot/interaction/PutRDF.java, htroot/interaction/Table.java, htroot/interaction_elements/Document_part.java, htroot/interaction_elements/Footer.java, htroot/interaction_elements/Loginstatus_part.java, htroot/interaction_elements/OverlayInteraction.java, htroot/interaction_elements/Tag_part.java, htroot/mediawiki_p.java, htroot/opensearchdescription.java, htroot/osm.java, htroot/rct_p.java, htroot/robots.java, htroot/sharedBlacklist_p.java, htroot/solr/select.java, htroot/ssitestservlet.java, htroot/suggest.java, htroot/test.java, htroot/www/welcome.java, htroot/yacy/crawlReceipt.java, htroot/yacy/hello.java, htroot/yacy/message.java, htroot/yacy/query.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacy/transferURL.java, htroot/yacy/urls.java, htroot/yacyinteractive.java, htroot/yacysearch.java, htroot/yacysearchitem.html, htroot/yacysearchitem.java, htroot/yacysearchlatestinfo.java, htroot/yacysearchtrailer.java, lib/dependencies.txt, lib/httpclient-4.2.1.License, lib/httpclient-4.2.1.jar, lib/httpcore-4.2.1.License, lib/httpcore-4.2.1.jar, lib/httpmime-4.2.1.License, lib/httpmime-4.2.1.jar, libbuild/GitRevTask/GitRevTask.java, libbuild/svnRevNr/svnRevNrParser.java, locales/cn.lng, locales/de.lng, locales/ru.lng, locales/uk.lng, nbproject/project.xml, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/Cache.java, source/de/anomic/crawler/CrawlProfile.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/NoticedURL.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/crawler/ResultImages.java, source/de/anomic/crawler/ResultURLs.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/RobotsTxtEntry.java, source/de/anomic/crawler/RobotsTxtParser.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/crawler/ZURL.java, source/de/anomic/crawler/retrieval/FTPLoader.java, source/de/anomic/crawler/retrieval/FileLoader.java, source/de/anomic/crawler/retrieval/HTTPLoader.java, source/de/anomic/crawler/retrieval/Request.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/crawler/retrieval/SMBLoader.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarkDate.java, source/de/anomic/data/BookmarkHelper.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/DidYouMean.java, source/de/anomic/data/ListManager.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/Translator.java, source/de/anomic/data/UserDB.java, source/de/anomic/data/WorkTables.java, source/de/anomic/data/wiki/AbstractWikiParser.java, source/de/anomic/data/wiki/WikiBoard.java, source/de/anomic/data/wiki/WikiCode.java, source/de/anomic/data/ymark/TablesRowComparator.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkCrawlStart.java, source/de/anomic/data/ymark/YMarkDate.java, source/de/anomic/data/ymark/YMarkJSONImporter.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/de/anomic/data/ymark/YMarkTables.java, source/de/anomic/data/ymark/YMarkXBELImporter.java, source/de/anomic/http/server/AugmentedHtmlStream.java, source/de/anomic/http/server/ChunkedInputStream.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/http/server/HTTPDProxyHandler.java, source/de/anomic/http/server/HTTPDemon.java, source/de/anomic/http/server/ServerSideIncludes.java, source/de/anomic/http/server/TemplateEngine.java, source/de/anomic/server/serverCore.java, source/de/anomic/server/serverCoreSocket.java, source/de/anomic/server/serverObjects.java, source/de/anomic/server/serverSwitch.java, source/de/anomic/tools/CryptoLib.java, source/de/anomic/tools/UPnP.java, source/de/anomic/tools/crypt.java, source/de/anomic/tools/cryptbig.java, source/net/yacy/YaCySearchClient.java, source/net/yacy/ai/example/ConnectFour.java, source/net/yacy/ai/example/Hanoi.java, source/net/yacy/ai/example/SchwarzerPeter.java, source/net/yacy/ai/greedy/Battle.java, source/net/yacy/ai/greedy/Context.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/cora/document/Classification.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/JSONTokener.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/document/RSSReader.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/cora/language/phonetic/Soundex.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/Vocabulary.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/DCTerms.java, source/net/yacy/cora/lod/vocabulary/DublinCore.java, source/net/yacy/cora/lod/vocabulary/Foaf.java, source/net/yacy/cora/lod/vocabulary/Geo.java, source/net/yacy/cora/lod/vocabulary/HttpHeader.java, source/net/yacy/cora/lod/vocabulary/Owl.java, source/net/yacy/cora/lod/vocabulary/Rdf.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/order/CloneableIterator.java, source/net/yacy/cora/order/CloneableMapIterator.java, source/net/yacy/cora/protocol/ByteArrayBody.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/Scanner.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/cora/services/federated/opensearch/SRURSSConnector.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/Schema.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SingleSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/services/federated/solr/SolrType.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/sorting/OrderedScoreMap.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/cora/storage/Files.java, source/net/yacy/cora/storage/HandleMap.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/cora/storage/SimpleARC.java, source/net/yacy/cora/storage/SizeLimitedMap.java, source/net/yacy/cora/storage/SizeLimitedSet.java, source/net/yacy/cora/util/SpaceExceededException.java, source/net/yacy/dbtest.java, source/net/yacy/document/AbstractParser.java, source/net/yacy/document/Autotagging.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SentenceReader.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/StringBuilderComparator.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/WordCache.java, source/net/yacy/document/WordTokenizer.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/content/dao/PhpBB3Dao.java, source/net/yacy/document/geolocation/GeoLocation.java, source/net/yacy/document/geolocation/GeonamesLocation.java, source/net/yacy/document/geolocation/Locations.java, source/net/yacy/document/geolocation/OpenGeoDBLocation.java, source/net/yacy/document/geolocation/OverarchingLocation.java, source/net/yacy/document/importer/MediawikiImporter.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHImporter.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/augment/AugmentParser.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/docParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/AbstractScraper.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/htmlParser.java, source/net/yacy/document/parser/images/bmpParser.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/odtParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/rdfParser.java, source/net/yacy/document/parser/rdfa/impl/RDFaParser.java, source/net/yacy/document/parser/rtfParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/sidAudioParser.java, source/net/yacy/document/parser/sitemapParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/xlsParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/gui/YaCyApp.java, source/net/yacy/gui/framework/Browser.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BLOB.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapColumnIndex.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/ObjectBuffer.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/citation/CitationReference.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/MetadataVocabulary.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/data/word/WordReference.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Cache.java, source/net/yacy/kelondro/index/Column.java, source/net/yacy/kelondro/index/Index.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/CachedFileReader.java, source/net/yacy/kelondro/io/CachedRecords.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/io/RandomAccessIO.java, source/net/yacy/kelondro/io/Records.java, source/net/yacy/kelondro/logging/ConsoleOutHandler.java, source/net/yacy/kelondro/logging/GuiHandler.java, source/net/yacy/kelondro/logging/Log.java, source/net/yacy/kelondro/logging/LogParser.java, source/net/yacy/kelondro/logging/LogalizerHandler.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/order/Base64Order.java, source/net/yacy/kelondro/order/Digest.java, source/net/yacy/kelondro/order/MergeIterator.java, source/net/yacy/kelondro/order/NaturalOrder.java, source/net/yacy/kelondro/order/RotateIterator.java, source/net/yacy/kelondro/order/StackIterator.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/AbstractReference.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/Index.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/IndexReader.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/rwi/TermSearch.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SQLTable.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/ByteArray.java, source/net/yacy/kelondro/util/ByteBuffer.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/GenerationMemoryStrategy.java, source/net/yacy/kelondro/util/MemoryControl.java, source/net/yacy/kelondro/util/MemoryStrategy.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/kelondro/util/StandardMemoryStrategy.java, source/net/yacy/kelondro/workflow/AbstractThread.java, source/net/yacy/kelondro/workflow/InstantBlockingThread.java, source/net/yacy/kelondro/workflow/InstantBusyThread.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/peers/dht/FlatWordPartitionScheme.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/peers/operation/yacySeedUploadFile.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/EventTracker.java, source/net/yacy/search/IndexingQueueEntry.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/search/solr/SolrServlet.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/HttpResponse.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/upnp/impls/InternetGatewayDevice.java, source/net/yacy/upnp/services/ISO8601Date.java, source/net/yacy/visualization/RasterPlotter.java, source/net/yacy/yacy.java, source/org/apache/tools/tar/TarInputStream.java

by Michael Peter Christen

- renamed DoubleSolrConnector to MirrorSolrConnector and added a
hit/miss/document cache to the MirrorSolrConnector.
- more abstraction to SolrDocument in Connector interface
- bugfixes in Solr field reader

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/query/SnippetProcess.java

by Michael Peter Christen

- added coordinate storage in solr schema
- fixed shutdown process
- fixed some solr-to-metadata reading
- added a large number of metadata attributes in ViewFile.html

Changed Files: defaults/solr/schema.xml, htroot/ViewFile.html, htroot/ViewFile.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/server/serverObjects.java, source/net/yacy/cora/document/ASCII.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by Michael Peter Christen

- Implemented and integrated the URIMetadataNode object which is a
metadata representation from the solr index. This shall replace metadata
from the built-in database in the future.
- added the Solr-driven metadata into the search index of YaCy which
makes it now possible to run YaCy without the old metadata index. This
is a major stept forward to a full migration to Solr.

Changed Files: defaults/solr.keys.list, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/de/anomic/crawler/ResultURLs.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/kelondro/data/word/WordReference.java, source/net/yacy/kelondro/data/word/WordReferenceRow.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/index/DocumentReference.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/ranking/ReferenceOrder.java, source/net/yacy/search/snippet/ResultEntry.java

by Michael Peter Christen

better abstraction for result writers using controlled vocabularies and
URIRefs

Changed Files: source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/lod/Vocabulary.java, source/net/yacy/cora/lod/vocabulary/CreativeCommons.java, source/net/yacy/cora/lod/vocabulary/DCTerms.java, source/net/yacy/cora/lod/vocabulary/DublinCore.java, source/net/yacy/cora/lod/vocabulary/Foaf.java, source/net/yacy/cora/lod/vocabulary/Geo.java, source/net/yacy/cora/lod/vocabulary/HttpHeader.java, source/net/yacy/cora/lod/vocabulary/Owl.java, source/net/yacy/cora/lod/vocabulary/Rdf.java, source/net/yacy/cora/lod/vocabulary/YaCyMetadata.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/SolrType.java, source/net/yacy/kelondro/data/meta/MetadataVocabulary.java

by Michael Peter Christen

refactoring

Changed Files: htroot/IndexFederated_p.java, htroot/api/schema_p.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/Schema.java, source/net/yacy/cora/services/federated/solr/ShardSelection.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by Michael Peter Christen

added two response writer for embedded solr interface:
a rss/opensearch writer and an enhanced solr xml writer.
The enhanced solr writer has less configuration overhead than the
original writer and should by slightly faster. The rss/opensearch writer
is at this time slightly incomplete compared with the already existing
rss search result form YaCy and also snippets are missing at this time.
To test the new interface, open for example:
http://localhost:8090/solr/select?wt=rss&q=olympia
The wt-code for the new result writers are=
wt=rss for opensearch
wt=exml for the enhanced solr xml writer.
Additionally, the SRU search parameters had been added to the solr
interface which can now also be used for a normal solr/xml search.

Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by orbiter

added more solr fields to integrate values from URIMetadataRow. All
writings to the Metadata-DB are now also done to solr. This includes
metadata transfer during search and rwi transfer.

The new/added solr fields are:

## time when resource was loaded
load_date_dt

## date until resource shall be considered as fresh
fresh_date_dt

## id of the host, a 6-byte hash that is part of the document id
host_id_s

## ids of referrer to this document
referrer_id_ss

## the md5 of the raw source
md5_s

## the name of the publisher of the document
publisher_t

## the language used in the document; starts with primary language
language_ss

## an external ranking value
ranking_i

## the size of the raw source
size_i

## number of links to audio resources
audiolinkscount_i

## number of links to video resources
videolinkscount_i

## number of links to application resources
applinkscount_i

Changed Files: defaults/solr.keys.list, htroot/IndexFederated_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/de/anomic/crawler/ResultURLs.java, source/de/anomic/crawler/retrieval/Response.java, source/net/yacy/cora/services/federated/solr/SolrDoc.java, source/net/yacy/cora/storage/ConfigurationSet.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/SolrField.java

by Michael Peter Christen

snippet retrieval loading processes may use a smaller minimum load time
value than crawling processes. This speeds up the search result
preparation dramatically.

Changed Files: htroot/Bookmarks.java, htroot/Crawler_p.java, htroot/DictionaryLoader_p.java, htroot/Load_RSS_p.java, htroot/ViewFile.java, htroot/ViewImage.java, htroot/api/getpageinfo.java, htroot/api/getpageinfo_p.java, htroot/api/webstructure.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/de/anomic/crawler/CrawlQueues.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/document/importer/OAIListFriendsLoader.java, source/net/yacy/document/importer/OAIPMHLoader.java, source/net/yacy/peers/graphics/OSMTile.java, source/net/yacy/peers/operation/yacyRelease.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java

by Michael Peter Christen

Abstraction of HandleMap and HandleSet

Changed Files: htroot/CrawlProfileEditor_p.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/Load_RSS_p.java, htroot/Table_API_p.java, htroot/Table_YMark_p.java, htroot/Tables_p.java, htroot/api/table_p.java, htroot/api/timeline.java, htroot/api/ymarks/add_ymark.java, htroot/api/ymarks/delete_ymark.java, htroot/api/ymarks/get_treeview.java, htroot/api/ymarks/import_ymark.java, htroot/yacy/search.java, htroot/yacy/transferRWI.java, htroot/yacysearch.java, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/Cache.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/NoticedURL.java, source/de/anomic/crawler/RSSLoader.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/ZURL.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarkDate.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/UserDB.java, source/de/anomic/data/WorkTables.java, source/de/anomic/data/wiki/WikiBoard.java, source/de/anomic/data/ymark/YMarkAutoTagger.java, source/de/anomic/data/ymark/YMarkTables.java, source/net/yacy/cora/storage/HandleMap.java, source/net/yacy/cora/storage/HandleSet.java, source/net/yacy/cora/util/SpaceExceededException.java, source/net/yacy/dbtest.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/interaction/Interaction.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/BLOB.java, source/net/yacy/kelondro/blob/Compressor.java, source/net/yacy/kelondro/blob/Heap.java, source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/blob/MapDataMining.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Stack.java, source/net/yacy/kelondro/blob/Stacks.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Cache.java, source/net/yacy/kelondro/index/Index.java, source/net/yacy/kelondro/index/IndexTest.java, source/net/yacy/kelondro/index/RAMIndex.java, source/net/yacy/kelondro/index/RAMIndexCluster.java, source/net/yacy/kelondro/index/RowCollection.java, source/net/yacy/kelondro/index/RowHandleMap.java, source/net/yacy/kelondro/index/RowHandleSet.java, source/net/yacy/kelondro/index/RowSet.java, source/net/yacy/kelondro/order/NaturalOrder.java, source/net/yacy/kelondro/rwi/AbstractIndex.java, source/net/yacy/kelondro/rwi/Index.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/kelondro/rwi/IndexReader.java, source/net/yacy/kelondro/rwi/ReferenceContainer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/rwi/ReferenceIterator.java, source/net/yacy/kelondro/rwi/TermSearch.java, source/net/yacy/kelondro/table/Relations.java, source/net/yacy/kelondro/table/SplitTable.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/NewsPool.java, source/net/yacy/peers/NewsQueue.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/dht/Dispatcher.java, source/net/yacy/peers/dht/FlatWordPartitionScheme.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java

by Michael Peter Christen

cleaned up classes and methods which are either superfluous at this time
or will be superfluous or subject of complete redesign after the
migration to solr. Removing these things now will make the transition to
solr more simple.

Changed Files: htroot/PerformanceQueues_p.html, htroot/PerformanceQueues_p.java, htroot/PerformanceQueues_p.xml, htroot/env/templates/submenuBlacklist.template, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/yacy.java

by Michael Peter Christen

Moved solr index-add method to the same method where the YaCy index is
written. Also done some code-cleanup.

Changed Files: htroot/IndexFederated_p.java, htroot/api/schema_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/IndexingQueueEntry.java, source/net/yacy/search/Shutdown.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/yacy.java

by Michael Peter Christen

extended abstraction of local and remote solr index using one front-end
for index administration and querying.

Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/DoubleSolrConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/SnippetProcess.java

by Michael Peter Christen

changed options in IndexFederated_p to switch on/off parts of the index
individually. The settings are experimental and the values of the
settings will be overwritten when an index migration from urldb to solr
starts.

Changed Files: defaults/yacy.init, htroot/IndexFederated_p.html, htroot/IndexFederated_p.java, htroot/yacysearchtrailer.java, source/net/yacy/kelondro/rwi/ReferenceContainerArray.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/yacy.java

by orbiter

- more abstraction for the RWI index as preparation for solr integration
- added options in search index to switch parts of the index on or off

Changed Files: htroot/Bookmarks.java, htroot/CrawlResults.java, htroot/IndexControlRWIs_p.html, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/api/yacydoc.java, htroot/yacy/urls.java, htroot/yacysearch.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/SitemapImporter.java, source/de/anomic/data/URLAnalysis.java, source/de/anomic/data/ymark/YMarkMetadata.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/kelondro/data/meta/URIMetadata.java, source/net/yacy/kelondro/data/meta/URIReference.java, source/net/yacy/kelondro/data/meta/URIReferenceNode.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/ResultEntry.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/yacy.java

by Michael Peter Christen

added a solr search index
- by default, a (empty) solr storage instance is created at
SEGMENTS/solr_36
- the index is written if in /IndexFederated_p.html the flag "embedded
solr search index" is switched on
- a standard solr query interface is available now with a new servlet at
http://127.0.0.1:8090/solr/select

To test this, do the following:
- switch to webportal mode
- switch on the feature as described
- do a crawl. this fills the solr index. The normal YaCy search will NOT
work now!
- do a solr query, like:
http://127.0.0.1:8090/solr/select?q=*:*
http://127.0.0.1:8090/solr/select?q=text_t:Help
play with different search fields as you can see in
/IndexFederated_p.html
You can use the standard solr query attributes as described in
http://wiki.apache.org/solr/SearchHandler

Changed Files: htroot/solr/select.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/server/serverObjects.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java, source/net/yacy/search/solr/SolrServlet.java

by Michael Peter Christen

collection of speed and memory saving hacks

Changed Files: htroot/api/yacydoc.java, source/de/anomic/crawler/ResultImages.java, source/de/anomic/crawler/RobotsTxt.java, source/de/anomic/crawler/ZURL.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/protocol/http/HTTPClient.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/blob/HeapWriter.java, source/net/yacy/kelondro/data/word/Word.java, source/net/yacy/kelondro/logging/ConsoleOutHandler.java, source/net/yacy/kelondro/logging/Log.java, source/net/yacy/kelondro/rwi/IODispatcher.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/visualization/RasterPlotter.java

by orbiter

reduced logging overhead (a bit)

Changed Files: source/net/yacy/document/AbstractParser.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/document/parser/pptParser.java, source/net/yacy/document/parser/psParser.java, source/net/yacy/document/parser/sevenzipParser.java, source/net/yacy/document/parser/swfParser.java, source/net/yacy/document/parser/tarParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/document/parser/vsdParser.java, source/net/yacy/document/parser/zipParser.java, source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/logging/Log.java

by orbiter

replaced more size() > 0 by !isEmpty()

Changed Files: htroot/CrawlStartScanner_p.java, htroot/IndexImportOAIPMH_p.java, htroot/Status.java, htroot/Table_API_p.java, htroot/yacysearchtrailer.java, source/de/anomic/crawler/Balancer.java, source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/retrieval/Request.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/net/yacy/ai/example/SchwarzerPeter.java, source/net/yacy/document/parser/mmParser.java, source/net/yacy/kelondro/rwi/IndexCell.java, source/net/yacy/search/index/SolrConfiguration.java

by orbiter

- replaced all length() == 0 and size() == 0 with isEmpty()
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods

Changed Files: htroot/BlacklistCleaner_p.java, htroot/Blacklist_p.java, htroot/Blog.java, htroot/BlogComments.java, htroot/Bookmarks.java, htroot/ConfigAccounts_p.java, htroot/ConfigNetwork_p.java, htroot/CrawlResults.java, htroot/CrawlStartScanner_p.java, htroot/Crawler_p.java, htroot/IndexControlRWIs_p.java, htroot/IndexControlURLs_p.java, htroot/Load_RSS_p.java, htroot/MessageSend_p.java, htroot/Network.java, htroot/SettingsAck_p.java, htroot/Status.java, htroot/Surftips.java, htroot/Table_API_p.java, htroot/Tables_p.java, htroot/ViewFile.java, htroot/Vocabulary_p.java, htroot/WatchWebStructure_p.java, htroot/WebStructurePicture_p.java, htroot/api/blacklists.java, htroot/api/blacklists_p.java, htroot/api/table_p.java, htroot/api/yacydoc.java, htroot/compare_yacy.java, htroot/yacy/message.java, htroot/yacy/search.java, htroot/yacysearchtrailer.java, libbuild/GitRevTask/GitRevTask.java, libbuild/svnRevNr/svnRevNrParser.java, source/de/anomic/crawler/CrawlProfile.java, source/de/anomic/crawler/CrawlSwitchboard.java, source/de/anomic/crawler/RobotsTxtEntry.java, source/de/anomic/crawler/RobotsTxtParser.java, source/de/anomic/crawler/retrieval/FTPLoader.java, source/de/anomic/crawler/retrieval/HTTPLoader.java, source/de/anomic/crawler/retrieval/Response.java, source/de/anomic/data/BlogBoard.java, source/de/anomic/data/BlogBoardComments.java, source/de/anomic/data/BookmarksDB.java, source/de/anomic/data/DidYouMean.java, source/de/anomic/data/MessageBoard.java, source/de/anomic/data/Translator.java, source/de/anomic/data/URLAnalysis.java, source/de/anomic/data/WorkTables.java, source/de/anomic/http/server/ChunkedInputStream.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/http/server/HTTPDemon.java, source/de/anomic/server/serverCore.java, source/de/anomic/server/serverObjects.java, source/de/anomic/tools/cryptbig.java, source/net/yacy/ai/example/Hanoi.java, source/net/yacy/cora/date/GenericFormatter.java, source/net/yacy/cora/date/ISO8601Formatter.java, source/net/yacy/cora/document/Classification.java, source/net/yacy/cora/document/JSONObject.java, source/net/yacy/cora/document/MultiProtocolURI.java, source/net/yacy/cora/document/RSSMessage.java, source/net/yacy/cora/language/phonetic/DoubleMetaphone.java, source/net/yacy/cora/language/phonetic/Metaphone.java, source/net/yacy/cora/language/phonetic/Soundex.java, source/net/yacy/cora/lod/JenaTripleStore.java, source/net/yacy/cora/lod/vocabulary/Tagging.java, source/net/yacy/cora/protocol/Domains.java, source/net/yacy/cora/protocol/HeaderFramework.java, source/net/yacy/cora/protocol/ResponseHeader.java, source/net/yacy/cora/protocol/ftp/FTPClient.java, source/net/yacy/cora/services/federated/solr/SingleSolrConnector.java, source/net/yacy/cora/sorting/ClusteredScoreMap.java, source/net/yacy/cora/storage/KeyList.java, source/net/yacy/dbtest.java, source/net/yacy/document/Autotagging.java, source/net/yacy/document/Condenser.java, source/net/yacy/document/Document.java, source/net/yacy/document/LibraryProvider.java, source/net/yacy/document/SnippetExtractor.java, source/net/yacy/document/StringBuilderComparator.java, source/net/yacy/document/TextParser.java, source/net/yacy/document/WordCache.java, source/net/yacy/document/content/DCEntry.java, source/net/yacy/document/content/dao/ImportDump.java, source/net/yacy/document/geolocation/GeonamesLocation.java, source/net/yacy/document/geolocation/Locations.java, source/net/yacy/document/geolocation/OpenGeoDBLocation.java, source/net/yacy/document/geolocation/OverarchingLocation.java, source/net/yacy/document/importer/ResumptionToken.java, source/net/yacy/document/parser/csvParser.java, source/net/yacy/document/parser/dwgParser.java, source/net/yacy/document/parser/genericParser.java, source/net/yacy/document/parser/html/ContentScraper.java, source/net/yacy/document/parser/html/ContentTransformer.java, source/net/yacy/document/parser/html/ScraperInputStream.java, source/net/yacy/document/parser/html/TransformerWriter.java, source/net/yacy/document/parser/images/genericImageParser.java, source/net/yacy/document/parser/ooxmlParser.java, source/net/yacy/document/parser/pdfParser.java, source/net/yacy/document/parser/torrentParser.java, source/net/yacy/document/parser/vcfParser.java, source/net/yacy/kelondro/blob/ArrayStack.java, source/net/yacy/kelondro/blob/BEncodedHeap.java, source/net/yacy/kelondro/blob/MapHeap.java, source/net/yacy/kelondro/blob/Tables.java, source/net/yacy/kelondro/data/meta/DigestURI.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/kelondro/data/word/WordReferenceFactory.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/kelondro/index/BufferedObjectIndex.java, source/net/yacy/kelondro/index/Column.java, source/net/yacy/kelondro/index/Row.java, source/net/yacy/kelondro/io/AbstractWriter.java, source/net/yacy/kelondro/io/CharBuffer.java, source/net/yacy/kelondro/logging/ThreadDump.java, source/net/yacy/kelondro/order/Base64Order.java, source/net/yacy/kelondro/rwi/AbstractReference.java, source/net/yacy/kelondro/rwi/ReferenceContainerCache.java, source/net/yacy/kelondro/table/Table.java, source/net/yacy/kelondro/util/ByteBuffer.java, source/net/yacy/kelondro/util/FileUtils.java, source/net/yacy/kelondro/util/ReverseMapIterator.java, source/net/yacy/kelondro/util/SetTools.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/NewsDB.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/peers/graphics/WebStructureGraph.java, source/net/yacy/peers/operation/yacySeedUploadFile.java, source/net/yacy/peers/operation/yacySeedUploadScp.java, source/net/yacy/repository/Blacklist.java, source/net/yacy/repository/FilterEngine.java, source/net/yacy/repository/LoaderDispatcher.java, source/net/yacy/search/EventTracker.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/query/AccessTracker.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java, source/net/yacy/search/ranking/RankingProfile.java, source/net/yacy/search/snippet/MediaSnippet.java, source/net/yacy/search/snippet/TextSnippet.java, source/net/yacy/upnp/Discovery.java, source/net/yacy/upnp/DiscoveryAdvertisement.java, source/net/yacy/upnp/DiscoveryListener.java, source/net/yacy/upnp/HttpResponse.java, source/net/yacy/upnp/devices/UPNPRootDevice.java, source/net/yacy/yacy.java

by sixcooler

bump to httpclient-4.2.1

Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/dependencies.txt, lib/httpclient-4.2.1.License, lib/httpclient-4.2.1.jar, lib/httpcore-4.2.1.License, lib/httpcore-4.2.1.jar, lib/httpmime-4.2.1.License, lib/httpmime-4.2.1.jar, nbproject/project.xml

by orbiter

refactoring of query attribute variable names for better consistency
with (next) stored query words

Changed Files: htroot/AccessTracker_p.java, htroot/yacy/search.java, htroot/yacysearch.java, htroot/yacysearchitem.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SearchEvent.java, source/net/yacy/search/query/SnippetProcess.java

by orbiter

small fixes

Changed Files: htroot/ConfigBasic.java, htroot/IndexControlRWIs_p.java, htroot/IndexCreateQueues_p.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/document/parser/swfParser.java

by orbiter

fixed concurrent query

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java

by orbiter

fixed generation of ontologies from index enumerations

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/search/index/Segment.java

by Michael Peter Christen

fix for NPE during host navigation computation

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java

by Michael Peter Christen

fixed GSA format

Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java

by Michael Peter Christen

gsa bugfix for date parser

Changed Files: htroot/gsa/searchresult.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java

by Michael Peter Christen

fixes for gsa result format

Changed Files: htroot/gsa/searchresult.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java

by Michael Peter Christen

fixed double-check

Changed Files: source/net/yacy/search/query/RWIProcess.java

by Michael Peter Christen

bugfix for remote search when search is done to solr

Changed Files: source/net/yacy/peers/Protocol.java

by Michael Peter Christen

fix for http://bugs.yacy.net/view.php?id=206

Changed Files: source/net/yacy/search/query/RWIProcess.java

by orbiter

fixes for putDocument and putMetadata

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java

by orbiter

fix for Index out of bounds exception in Network servlet

Changed Files: htroot/Network.java

by Lotus

fix xss bug #204

Changed Files: htroot/Network.java

by Michael Peter Christen

bad hack to prevent a bug appearing in solr

Changed Files: defaults/solr/solrconfig.xml, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by Michael Peter Christen

fix for a NPE

Changed Files: source/net/yacy/repository/Blacklist.java

by Michael Peter Christen

bugfix for a NPE

Changed Files: source/net/yacy/peers/Protocol.java

by Michael Peter Christen

fixed node type calculation for principal peers

Changed Files: source/net/yacy/peers/Protocol.java

by Michael Peter Christen

fix for http://bugs.yacy.net/view.php?id=202

Changed Files: htroot/ViewFile.java, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java

by orbiter

fix for RSS reader

Changed Files: source/net/yacy/cora/document/RSSReader.java

by orbiter

release 1.1

Changed Files: build.properties

by orbiter

added a direct access to solr search api to enhance the visibility if
the embedded solr

Changed Files: htroot/env/templates/header.template

by orbiter

Merge commit 'c2341a175fdd755a34965ff63c7ea437b380352d'

Changed Files: source/net/yacy/cora/protocol/ftp/FTPClient.java

by David Rubio

Fixed a bug that prevented Yacy from indexing files with non ASCII filenames in FTP servers.

Previously Yacy could read file listings in UTF-8, but couldn't send commands to the FTP server in UTF-8 (the second byte of every multi-byte character was ignored), which caused a lot of errors on the server side.
Now it handles UTF-8 correctly.

Changed Files: source/net/yacy/cora/protocol/ftp/FTPClient.java

by orbiter

omit xml parsing when using the embedded solr server

Changed Files: source/net/yacy/cora/services/federated/solr/EmbeddedSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java

by orbiter

added the
QueryResponse query(SolrParams params)
method to the SolrServerConnector which is necessary to use facets in
solr search.

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java

by orbiter

redesign of YaCySchema and SolrDoc handling

Changed Files: source/de/anomic/crawler/ZURL.java, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/YaCySchema.java

by orbiter

refctoring

Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EmbeddedSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServlet.java, source/net/yacy/search/index/Fulltext.java

by Michael Peter Christen

log queries anonymous from gsa+solr requests

Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/query/AccessTracker.java

by Michael Peter Christen

added snippet computation to solr/rss and gsa result writer

Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java, source/net/yacy/search/query/SnippetProcess.java

by Michael Peter Christen

- reduced memory usage in index transmission using a transformation of
Node to Row objects
- removed peerDeparture in solr remote search in case that peer does not
answer (this may be normal because it is allowed to switch this off)

Changed Files: htroot/IndexControlRWIs_p.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/dht/Transmission.java, source/net/yacy/search/index/Fulltext.java

by Michael Peter Christen

re-activated audio and video search because they obviously work (!)

Changed Files: defaults/yacy.init

by Michael Peter Christen

corrected solr query syntax

Changed Files: source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/QueryParams.java

by Michael Peter Christen

- enhanced caching after search queries to solr
- reduced caching after short memory

Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java

by orbiter

sorted the solr schema into mandatory and optional fields; reduced
number of used field to reduce solr index size

Changed Files: defaults/solr.keys.list, source/net/yacy/search/index/YaCySchema.java

by orbiter

fix from gaston in
http://forum.yacy-websuche.de/viewtopic.php?p=26909#p26909

Changed Files: source/net/yacy/search/index/SolrConfiguration.java

by orbiter

removed unused classes

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadata.java

by Michael Peter Christen

added authorization-based maximum results limitation to solr and gsa
search

Changed Files: htroot/gsa/searchresult.java, htroot/solr/select.java

by Michael Peter Christen

added gzip encoding to solr2sor http interface, client side (server
already works)

Changed Files: source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java

by Michael Peter Christen

added a tooltip for search navigation to mention that search pages can
be navigated using the TAB key

Changed Files: htroot/js/yacysearch.js

by Michael Peter Christen

gsa format update

Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/query/QueryParams.java

by Michael Peter Christen

more abstraction and less parameter overhead for remote search

Changed Files: htroot/yacy/search.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/graphics/NetworkGraph.java, source/net/yacy/search/query/SearchEvent.java

by Michael Peter Christen

code simplifications

Changed Files: htroot/Connections_p.java, source/de/anomic/server/serverSwitch.java, source/net/yacy/interaction/AugmentHtmlStream.java, source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/SearchEvent.java

by Michael Peter Christen

removed strange assert statements and simplified code in metadata
transformation

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/data/meta/URIMetadataRow.java

by orbiter

refactoring in remote search and stub for remote node peer selection

Changed Files: source/net/yacy/peers/Protocol.java, source/net/yacy/peers/RemoteSearch.java, source/net/yacy/peers/Seed.java, source/net/yacy/peers/dht/PeerSelection.java, source/net/yacy/search/query/SearchEvent.java

by orbiter

- get nice text_t values from metadata conversions that are stored into
solr as fulltext search index.
- added slow migration from old metadata to solr index entries: each
entry from the old metadata is removed from that data structure and
written into solr.

Changed Files: source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/SolrConfiguration.java

by orbiter

reduced sleep times

Changed Files: source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java

by orbiter

added ramaining iteration methods for solr in fulltext class

Changed Files: htroot/IndexControlURLs_p.java, source/net/yacy/cora/order/CloneableIterator.java, source/net/yacy/search/index/Fulltext.java

by orbiter

hack to removed StringBuilder overhead in query construction

Changed Files: source/net/yacy/cora/services/federated/solr/SolrServerConnector.java

by orbiter

reduced solr cache sizes to check if that solves memory problems a bit

Changed Files: defaults/solr/solrconfig.xml

by orbiter

explicit double-check in transferURL

Changed Files: htroot/yacy/transferURL.java

by orbiter

reverted bf55f6917652909f8eb465ccefd1f7ccb4c4d364
to have a fall-back option in case that memory problems as reported in
http://forum.yacy-websuche.de/viewtopic.php?p=26901#p26901
for full-solr installation are too strong and we have to work with an
'small memory footprint' peer system.

Changed Files: source/net/yacy/search/index/Fulltext.java

by Michael Peter Christen

added concurrent iterator methods to the solr connectors

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrServerConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by Michael Peter Christen

refactoring

Changed Files: htroot/IndexFederated_p.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RemoteSolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java

by Michael Peter Christen

better check for bad urls in url transmission

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataRow.java

by Michael Peter Christen

added deleteByQuery to solr connectors

Changed Files: .classpath, source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java, source/net/yacy/cora/services/federated/solr/MultipleSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/cora/services/federated/solr/ShardSolrConnector.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java

by Michael Peter Christen

removed write methods to old metadata file type; all metadata now goes
to solr

Changed Files: source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java

by Michael Peter Christen

refactoring

Changed Files: htroot/IndexControlURLs_p.java, htroot/YBRFetch_p.java, source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/search/index/Fulltext.java, source/net/yacy/search/index/Segment.java, source/net/yacy/search/query/RWIProcess.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/ranking/BlockRank.java

by Michael Peter Christen

upgrade to solr 3.6.1

Changed Files: .classpath, addon/YaCy.app/Contents/Info.plist, build.xml, lib/apache-solr-core-3.6.1.License, lib/apache-solr-core-3.6.1.jar, lib/apache-solr-solrj-3.6.1.License, lib/apache-solr-solrj-3.6.1.jar

by Michael Peter Christen

bugfix for solr connector, possibly a cause for
http://forum.yacy-websuche.de/viewtopic.php?p=26893#p26893

Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java

by Michael Peter Christen

enhanced snippet fetch - removed a bug that caused documents to be
parsed even if a solr text was available

Changed Files: source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/search/query/SnippetProcess.java, source/net/yacy/search/snippet/TextSnippet.java

by cominch

local robots.txt: disallow external crawlers to follow the URL proxy

Changed Files: htroot/robots.txt

by Michael Peter Christen

using the solr search index to concurrently search within solr and the
rwis during local search requests.

Changed Files: source/net/yacy/kelondro/data/word/WordReferenceVars.java, source/net/yacy/search/query/QueryParams.java, source/net/yacy/search/query/RWIProcess.java

by Michael Peter Christen

added clear-text search words in query params

Changed Files: htroot/yacy/search.java, htroot/yacysearch.java, source/net/yacy/search/query/QueryParams.java

by Michael Peter Christen

- added a content-encoding: gzip to streamed http server responses
- finish and close streamed http responses immediately
- this applies only to the solr interface which should be much faster
now!

Changed Files: source/de/anomic/http/server/HTTPDFileHandler.java

by Michael Peter Christen

doctype2mime fix, influences metadata conversion between old metadata
and solr

Changed Files: defaults/httpd.mime, source/de/anomic/crawler/retrieval/Response.java, source/net/yacy/cora/document/Classification.java

by cominch

changed local robots.txt to prevent external crawlers to submit random
search queries

Changed Files: htroot/robots.txt

by Michael Peter Christen

more attempts to clean the index (cleaning is faster then)

Changed Files: source/net/yacy/peers/Network.java

by Michael Peter Christen

fixed some peer-ping connection details
- larger time-out
- removed too old seedlist
- fixed a bug in connection test

Changed Files: defaults/yacy.network.freeworld.unit, htroot/ViewProfile.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/peers/Network.java, source/net/yacy/peers/Protocol.java

by orbiter

addon to e74d66e28cce7b9674ad5011e5db7970ccaf5635
(removed htmlparser.jar): for Mac App

Changed Files: addon/YaCy.app/Contents/Info.plist

by Michael Peter Christen

replaced yacy xml encoding by solr xml encoding

Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java

by Michael Peter Christen

enhanced GSA and RSS output format: corrected date, added some missing
fields, added xml encoding for utf8

Changed Files: source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java

by Michael Peter Christen

added a very rudimentary, incomplete, non-verified GSA response writer
for solr. Try this:
http://localhost:8090/gsa/searchresult?q=pdf&site=col1&num=10

Changed Files: .classpath, htroot/gsa/searchresult.java, htroot/solr/select.java, source/de/anomic/server/serverObjects.java, source/net/yacy/cora/services/federated/solr/GSAResponseWriter.java, source/net/yacy/cora/services/federated/solr/SolrConnector.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by Michael Peter Christen

- added xslt support for solr result formats.
try i.e.
http://localhost:8090/solr/select?q=*:*&start=0&rows=10&wt=xslt&tr=json.xsl
- added servlet-side mime-type configuration for streamed servlets. this
is used for the result formatters in solr result formats

Changed Files: defaults/solr/xslt/example.xsl, defaults/solr/xslt/json.xsl, htroot/solr/select.java, source/de/anomic/http/server/HTTPDFileHandler.java, source/de/anomic/server/serverCore.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by cominch

augmented browsing: remove htmlparser library

Changed Files: .classpath, build.xml

by cominch

augmented browsing: replace htmlparser by jsoup, which is more stable
and reliable

Changed Files: .classpath, build.xml, defaults/yacy.init, lib/jsoup-1.6.3.jar, source/net/yacy/interaction/AugmentHtmlStream.java

by cominch

added a possibility to define a custom network definition URL for remote
management

Changed Files: htroot/ConfigNetwork_p.html, htroot/ConfigNetwork_p.java

by Michael Peter Christen

ups

Changed Files: source/net/yacy/cora/services/federated/solr/MirrorSolrConnector.java

by Michael Peter Christen

another fix to the Solr metadata reading process and to the shutdown
process

Changed Files: source/net/yacy/kelondro/blob/HeapModifier.java, source/net/yacy/kelondro/data/meta/URIMetadataNode.java, source/net/yacy/kelondro/workflow/WorkflowProcessor.java

by Michael Peter Christen

removed unused class

Changed Files: source/net/yacy/search/index/MetadataRepository.java

by orbiter

tried to prevent calls to bad-hack getSize() method and reduced overhead
of that method a bit.

Changed Files: source/net/yacy/cora/services/federated/solr/AbstractSolrConnector.java, source/net/yacy/cora/services/federated/solr/RetrySolrConnector.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/solr/EmbeddedSolrConnector.java

by orbiter

patch from hint in
http://forum.yacy-websuche.de/viewtopic.php?p=26858#p26858
from gaston

Changed Files: htroot/yacy/transferRWI.java

by orbiter

changed behaviour of metadata storage: in case that any solr is
attached, the metadata is not written to the metadata-db, even if it is
enabled but instead to solr. This prevents that metadata is written in
two store systems at the same time. It is also the next step to migrate
the current metadata-db to solr.

Changed Files: htroot/IndexControlRWIs_p.java, htroot/yacy/crawlReceipt.java, htroot/yacy/transferURL.java, source/net/yacy/peers/Protocol.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java

by orbiter

removed unused classes

Changed Files:

by Michael Peter Christen

more abstraction of the YaCySchema -> Opensearch matching process

Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/EnhancedXMLResponseWriter.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java

by Michael Peter Christen

more abstraction for solr query params parsing

Changed Files: htroot/solr/select.java

by Michael Peter Christen

set the title every time, it is possible that it has changed

Changed Files: htroot/solr/select.java, source/net/yacy/cora/services/federated/solr/OpensearchResponseWriter.java

by ????????? ?????? ?????????

Fix an error in Russian translation: "can not" => "can".

Changed Files: locales/ru.lng

by Michael Peter Christen

replaced the multivalue generic string field name suffix _ss by _txt
because _ss is not part of the standard solr example schema.

Changed Files: defaults/solr.keys.list, source/net/yacy/search/index/SolrConfiguration.java, source/net/yacy/search/index/SolrField.java

by Michael Peter Christen

added the JSON response writer to solr interface, add &wt=json to the
servlet GET properties to use this format

Changed Files: htroot/solr/select.java

by sixcooler

prevent merge of blobs that can't be handled in memory

Changed Files: source/net/yacy/kelondro/blob/ArrayStack.java

by Michael Peter Christen

nowrap from gaston in forum
http://forum.yacy-websuche.de/viewtopic.php?p=26815#p26815

Changed Files: htroot/Network.html

by sixcooler

check content domain fix:
search image/media should not show pages containing image/media
search text should show all/text but image/media

Changed Files: source/net/yacy/search/query/RWIProcess.java

by sixcooler

close augmented stream if filled from cache to get its content
use augmented stream if proxyAugmentation is set only

Changed Files: source/de/anomic/http/server/HTTPDProxyHandler.java

by Michael Peter Christen

better calculation of possible saving in HeapReader index data structure

Changed Files: source/net/yacy/kelondro/blob/HeapReader.java

by Michael Peter Christen

documentation/comments

Changed Files: source/net/yacy/kelondro/blob/HeapReader.java, source/net/yacy/kelondro/index/HandleMap.java

by sixcooler

no translation of queue-links

Changed Files: locales/cn.lng, locales/de.lng, locales/ru.lng, locales/uk.lng

by Michael Peter Christen

cleanup

Changed Files: htroot/yacysearch.java, source/de/anomic/crawler/CrawlQueues.java, source/net/yacy/cora/protocol/RequestHeader.java, source/net/yacy/search/Switchboard.java

by Michael Peter Christen

added user-authentication protection to solr search (same as implemented
for yacysearch)

Changed Files: htroot/solr/select.java

by Michael Peter Christen

better explain how to access the embedded solr

Changed Files: htroot/IndexControlRWIs_p.html, htroot/IndexFederated_p.html

by reger

removed localized number formatting from num-results_totalcount response (this is only used in xml and json where localized format is not valid)

Changed Files: htroot/yacysearch.java, htroot/yacysearchtrailer.java

by orbiter

patches to ensure that solr connectors are active ony if they have a
solr object assigned and vice versa

Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java

by orbiter

embedded solr is only initiated if it is activated with
IndexFederated_p.html

Changed Files: htroot/IndexFederated_p.java, htroot/solr/select.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/index/DocumentIndex.java, source/net/yacy/search/index/MetadataRepository.java, source/net/yacy/search/index/Segment.java, source/net/yacy/yacy.java

by Michael Peter Christen

source change in classpath

Changed Files: .classpath

by Lotus

partial html fix for
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4454

Changed Files: htroot/yacysearchitem.html

by orbiter

added classpath for htroot/solr

Changed Files: .classpath

by Michael Peter Christen

allow larger log entries

Changed Files: source/net/yacy/kelondro/logging/Log.java

by Michael Peter Christen

removed a crawler overhead (terminated loop which searches greatest
stack that has zero-waiting urls). This should cause a slightly faster
crawl for crawl stacks with many different domains in the crawl queue.

Changed Files: source/de/anomic/crawler/Balancer.java

by Michael Peter Christen

enhancement in internal data organization which should generate less
synchronizations in database access

Changed Files: source/net/yacy/kelondro/io/BufferedRecords.java, source/net/yacy/kelondro/io/Records.java

by orbiter

less usage of generic logger to avoid logger generation overhead

Changed Files: htroot/yacysearch.java, source/net/yacy/peers/PeerActions.java, source/net/yacy/peers/SeedDB.java, source/net/yacy/search/query/SnippetProcess.java

by orbiter

prevent enqueueing of non-loggeable logging entries

Changed Files: source/net/yacy/kelondro/logging/Log.java

by Michael Peter Christen

reduction of logging to prevent too much IO caused be logging

Changed Files: source/de/anomic/crawler/CrawlStacker.java, source/de/anomic/crawler/ZURL.java, source/net/yacy/search/Switchboard.java

by Michael Peter Christen

fixed a memory leak inside the logger which appeared if the log was
writter faster that the logger is able to print this out to its out
stream. A very large collection of unwritten log outputs had been seen
during strong crawling. The new ArrayBlockingQueue is limited to prevent
this case.

Changed Files: source/net/yacy/kelondro/logging/Log.java

by Michael Peter Christen

added creation of subpath pattern when crawl start is 'from file'

Changed Files: htroot/Crawler_p.java

by orbiter

fix for url matcher of multiple amp& in an url, see:
http://forum.yacy-websuche.de/viewtopic.php?f=8&t=4439&p=26650#p26650

Changed Files: source/net/yacy/cora/document/MultiProtocolURI.java

by Roland 'Quix0r' Haeder

- removed cleaning of blacklist cache on startup
- added cleaning of blacklist cache if cache is modified in interface
- extended cache saving to all cache types
- moved cache location to DATA/LISTS
- fixed static file path which was relative to the application path but
should be relative to data path - which is different in debian and mac
implementations

Changed Files: bin/localsearch.sh, htroot/Blacklist_p.java, source/de/anomic/data/ListManager.java, source/net/yacy/repository/Blacklist.java

by orbiter

using SwitchboardConstants for solr attributes

Changed Files: htroot/IndexControlRWIs_p.java, htroot/IndexFederated_p.java, source/net/yacy/search/Switchboard.java, source/net/yacy/search/SwitchboardConstants.java, source/net/yacy/search/query/QueryParams.java